Challenge 1 Resourses
Challenge 1 submissions may request up to six 24-core nodes with 64 GiB of memory (a total of 144 cores), the same nodes and interconnect described below. The default number of nodes is 1.
To request more than 1 node, put the following in submission.conf as described on the Languages page.
srun -N x where x is a positive integer less than or equal to 6.
The Evaluation Platform hardware consists of a multi-node cluster with a Mellanox 4x FDR Infiniband interconnect (54.54 Gigabits/second data bandwidth; 4.5 GB/s possible with block sizes >6 million double words) in a 2-1 oversubscribed fat tree. Sets of 24 nodes have 12 uplinks into the large core Infiniband switch. Each node has 64 GiB of 4-channel 2133 MHz DDR4 SDRAM (68 Gigabytes/second) memory; two Intel Xeon E5-2670 v3 (Haswell) CPUs (Turbo boost NOT enabled), each with 12 cores (24 cores per node) and a clock speed of 2.30 GHz (peak floating point performance per node is 883.2 GFlops/sec); and 480 GB Intel® SSD 530 Series disk drives with SATA 3.0 6 Gbit/sec interfaces (sequential read (up to)540 MB/s, sequential write (up to)490 MB/s).
Currently, an Entrant may use up to 6 nodes for an MPI submission. Please see the submission.conf section on the Languages page for details. Nodes will be under the sole control of the Entrant and not shared with other users. The hardware configuration of all nodes is identical.
Results from each submission are gzipped and placed in tar files on dtn2.pnl.gov, an externally-facing gateway for transferring large amounts of data efficiently. It is connected to the Internet via multiple 10 gigabit per second links to the Pacific Northwest Gigapop and the Seattle Internet Exchange. It is not uncommon to see transfer rates over 1 gbps to various sites around the world.
The Evaluation Platform software uses the Red Hat Enterprise Linux Workstation release 6.6 (Santiago) operating system (kernel-2.6.32-642.6.2.el6.x86_64). Builds using CentOS6 (6.6 or greater preferred), CentOS7 and Red Hat 7 are fine. Ubuntu is not supported.
Requests for additional software will be considered. Please include requests on your registration form or send them to the GO Operations Team with subject line "Request for Additional Software".
Using the directive "srun_options = -p gpu -N 1" in submission.conf, it is possible to access 8 NVidia Tesla K-80 GPU nodes with the same basic hardware configuration as the evaluaiton nodes. Use of GPUs is for demonstration purposes only for now.
Other GPUs are potentially available, but since they break the uniform evaluation model the Competition is striving for there is no assurance that such requests would be granted for other than demonstration purposes. Send requests to the GO Operations Team with subject line "GPU request".
The other GPUs all use dual Intel Broadwell E5-2620 v4 @ 2.10GHz CPUs with 16 cores per node, 64 GB 2133Mhz DDR4 memory per node (4 nodes with 384 GB of memory), with disk, interconnect, and OS the same at the Evaluation Platform nodes.There are 31 nodes with NVIDIA P100 12GB PCI-e based GPUs, 6 nodes with dual NVIDIA V100 16GB PCI-e based GPUs, and one NVIDIA DGX-2.
As mentioned above, MPI may be used with up to six 24-core nodes (144 cores total). See the submission.conf section of the Languages page for details.
If you have any questions or concerns regarding the evaluation platform, please contact the GO Operations Team with subject line "Evaluation Platform".