What is INCITE? Each year the Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program awards to researchers billions of supercomputer processor hours and 100 trillion bytes of data storage space at the Department of Energy’s (DOE’s) leadership computing facilities (LCFs) for unclassified supercomputing, which include some of the most powerful computers in the world. The program seeks computationally intensive, large-scale research projects that can make high-impact scientific and engineering advances through the use of a substantial allocation of computer time and data storage. The INCITE program specifically encourages proposals from research organizations including universities, national laboratories, and industry.
What is new in INCITE this year?
The Computational Readiness Review has been renamed the Technical Assessment. This name change is to clarify the role the assessment plays in the INCITE review process.
PIs are being asked to submit a 250-300 word abstract when submitting their proposals. If a proposal is awarded, this abstract will be used as a first draft to develop a public fact sheet.
Who manages INCITE? The INCITE program is jointly managed by the LCF centers at Argonne National Laboratory (ANL) and Oak Ridge National Laboratory (ORNL). The peer review of proposals is carried out by external groups of experts from national laboratories, universities, and industry; National Academy members; and senior computational science researchers who have a working knowledge of the current computational challenges and opportunities in their fields.
Who should I contact if I have questions about the INCITE program, the proposal template, or my application? Questions about the INCITE program and applications should be directed to the INCITE manager at [email protected]. The INCITE manager may forward your query to a representative of the Argonne Leadership Computing Facility (ALCF) or Oak Ridge Leadership Computing Facility (OLCF) as appropriate.
When is the next call for proposals? The INCITE call for proposals is issued annually in the spring. Awards are made for the following calendar year. However, the LCFs have a certain amount of additional system time available at the discretion of the center director throughout the year. For information about these smaller Director’s Discretionary allocations and center contact information, Contact Us.
Can you extend the deadline for proposal submissions?The INCITE Call for Proposals is from mid-April through the end of June (see the call for proposals for exact dates). The INCITE manager will consider extending the deadline if PIs experience documented difficulties submitting a proposal. However, researchers should begin the submission process as early as possible under the assumption that the deadline will not be extended. Proposals may be initiated and saved, allowing the PI to begin uploading content prior to the deadline.
Who is eligible to participate in INCITE? The INCITE program is open to US- and non-US-based researchers and research organizations needing large allocations of computer time, supporting resources, and data storage to pursue transformational advances in science and engineering. INCITE considers requests regardless of a PI’s funding source (ex. DOE, NSF, state, private, etc.) The intent of INCITE is to support large-scale, computationally intensive projects that would not be possible or productive without petascale computing. Applicants must present evidence that their proposed production simulations can make effective use of a significant fraction, in most cases 20% or more, of the high-performance computing (HPC) systems offered for allocation. Applicants planning to execute ensemble jobs are advised to review the Frequently Asked Question “Can I meet the computationally intensive criterion by loosely coupling my jobs?”
Must I have research funding from the Department of Energy? No, DOE sponsorship is not required. However, researchers are expected to have all funding and any other elements necessary to ensure the success of their research project in place at the time of their application.
Are researchers and teams from other federal agencies eligible to apply? Yes, research applications from other agencies are accepted.
Are foreign researchers and teams eligible to apply? Yes, their proposals will be evaluated on the same scientific and technical criteria as those of domestic researchers. U.S. collaborations are encouraged, but not required.
Does the proposal have to come from a team? No, the solicitation is open to individual researchers, a team of researchers from the same institution, or multi-institutional teams.
Are teams based around a community-maintained code or end stations eligible to apply? Yes. If the research objectives of a proposal are best satisfied by the operation of a community-maintained code or suite of codes on INCITE resources, such use is entirely acceptable. If the requested system time is to be part of a long-term community activity, this should be noted in the proposal.
Does INCITE accept proprietary proposals? Yes. The peer review selection process and project reporting requirements describe within the INCITE Overview and Polices will be implemented for both proprietary and non-proprietary user proposals. Individuals considering submittal of a proposal for proprietary research must contact the INCITE manager, [email protected] before the call for proposal closes to discuss the policy on proprietary work.
Does the research that I propose have to be in line with the Department of Energy mission, or have DOE research collaborators? No, alignment with the DOE mission is not required. Collaborators funded by DOE or located at DOE national laboratories are not required.
I previously received an INCITE award. Am I eligible to apply again? Yes. A previous recipient of an INCITE award, either a one-year or multi-year award, may submit a new proposal.
What questions are asked of the reviewers about my proposal?Details about the reviewer questions and ratings can be found on the proposal template web page.
How are INCITE projects chosen? IINCITE proposals are subjected to two reviews: (1) computational readiness and (2) peer review by an international panel of experts. Awards are based on the quality and impact of the research and the suitability of the proposed simulations for the requested resource. In the computational-readiness review, each LCF evaluates all qualified proposals for the readiness and scalability of the code and its algorithms. Experts are drawn from the facility staff and other institutional personnel who are well versed in the unique requirements of the leadership-class systems as well as experts from the computational science community, as needed. Reviewers are given the opportunity to submit a list of questions to the project’s PI to clarify vague or incomplete proposal information. In the peer review, the INCITE program solicits independent computational science, engineering, and computer science peer reviewers to evaluate each proposal’s potential for scientific or technical impact. Proposals are evaluated on scientific/technical merit, expected impact, qualification of the PI, qualification and composition of the proposed research team, and the computational plan. Review panels are composed of application domain experts from national laboratories, universities, and industry; National Academy members; and senior computational science researchers who have a working knowledge of the current computational challenges and opportunities in their fields. The INCITE Awards Committee, composed of the LCF management teams, makes selections based on the rankings by the peer-review panel. The readiness ratings are used to determine the capability of each project to effectively use the selected system and are based on proficiency shown through benchmarking data and/or proposed development plans. Proposals are selected that answer a high-impact, key scientific/technical question and/or point to a new area of research. Successful proposal teams demonstrate a very clear understanding of petascale computing and can optimally use these resources to accomplish the stated scientific/technical goals.
When will I hear if I got a award? Announcements will be made in early November. All PIs of new and renewed proposals will be contacted by the INCITE manager at that time.
What is Early Career Track? The INCITE Program created the Early Career Track to encourage researchers establishing their research careers. INCITE is committing 10% of allocatable time to meritorious projects.
Will the Early Career Track increase my chances of receiving an award? Potentially, this could increase chances of an award. Projects must still be deemed scientifically meritorious through the review process INCITE uses each year.
What do I need to do to be considered on the Early Career Track? If you have not been a previous INCITE PI, in the application process, select ‘Yes’ at ” If you are within 10 years of your PhD, would you like to be considered in the Early Career Track?” You will need to write a paragraph about how the INCITE proposal fits into your five-year, and research, and career goals.
What review criteria will be used for the Early Career Track? The same criteria for computational readiness and scientific merit will be applied to projects in the Early Career Track, as will be applied to projects in the traditional track. the difference will be manifest in awards decisions by the INCITE management committee.
What computing resources does INCITE use? DOE is the nation’s largest supporter of basic research in the physical sciences and provides a portfolio of LCFs housing some of the world’s most advanced supercomputers. It makes several of its supercomputers available for use in tackling large-scale, high-potential research projects in a broad array of science and engineering domains that are judged to be the most scientifically or technically promising in the INCITE peer-review process. For a description of the current systems and any anticipated new systems or system modifications that may be relevant to multiyear proposals, see the INCITE HPC Resources page.
Can I submit a proposal to use resources at both facilities? The applicant can request one or more resources. If allocations on more than one resource are requested, justification for the use of each resource must be clearly stated in the Description of Research and Computational Readiness sections of the application. Relevance to the accomplishment of the stated research goals should be the principal determining factor in any resource request.
If I get an INCITE award, how do I get funding for my research labor and other items needed to carry out the INCITE work? The INCITE award is for allocation of computer time and data-storage resources only. Applicants must have all other resources required for the successful completion of the INCITE work committed to them at the time of their INCITE application. Applicants who cannot demonstrate compliance with this requirement will be deemed ineligible for an INCITE award.
What staff support would an INCITE project receive from the LCFs? INCITE key projects have an assigned expert, the ‘catalyst’, to maximize and accelerate research. Catalysts are computational scientists that are experts in their fields: computational chemistry, physics, fluid dynamics, astrophysics, data analytics, workflows, machine/deep learning etc., and actively participate in the research projects. Projects also benefit from the support provided by the scientific and operational staff of the facility. This includes support from performance engineers, as well as visualization and operations team members, who are all available to help the project as needed.
What resources will be allocated in the INCITE program in 2024? Frontier is now available. The best path to establishing compute time requirements on Frontier is by running on Frontier, by requesting a Director’s Discretionary allocation (link). If this is not possible then the next best alternative is to utilize scaling data from a large GPU machine combined with limited performance data from an AMD GPU (ideally MI250X). If this is not possible, then scaling data from a large GPU machine (e.g. Summit) with projections to the MI250X based on the GPU specifications may be used. If performance data from a MI250X is not used, the authors should include a discussion on the extrapolation used to estimate time on Frontier. All hours requested must be in Frontier-native node hours.
For a three-year proposal, how do I estimate my resource request on Frontier (OLCF) in 2024 For the 2025 INCITE proposal submission, Frontier allocations in will be requested in “Frontier node-hours.” For planning purposes for this cycle, we expect the average INCITE project will be awarded approximately 1 million Frontier node-hours in 2024.
For a three-year proposal, how do I estimate my resource request on Aurora (ALCF) in 2024? Estimates for Aurora should be made on the basis of comparable GPU-based machines, in Aurora node hours; reasonable estimates could be derived from systems that include multiple GPUs per node. State of the art GPUs such as NVIDIA A100 or AMD MI200 provide a good foundation from which to estimate performance and therefore compute-time on the Intel PVC GPUs that will be deployed for Aurora. Polaris, with four A100 GPUs per node is, therefore, a good basis for estimating performance on Aurora.
PIs from ESP or ECP projects who have been running their application on pre-Aurora hardware should indicate that in their proposal, and specify the programming model used to leverage the GPUs; performance details on these systems should, however, not be included in the INCITE proposal due to their potentially sensitive nature, but instead work with their collaborator in the ALCF.
What constitutes a “computationally intensive” research project? A computationally intensive research project refers broadly to computing and/or data projects which would require a significant fraction of the processing resources and systems at the facility (in most cases 20% or more) for the proposed production science runs. Data-centric and machine-learning/analytics/AI based applications should utilize a significant fraction of the resources but may have a different usage profile across processor, memory, network, or disk usage. Please refer to the Frequently Asked Question “What constitutes a Data and/or AI research project?” for additional guidance. The limited history but dramatic growth and potential impact of data and AI oriented applications running at capability scale on LCF resources will be considered when evaluating the application readiness of the proposals.
Can I meet the computationally intensive criterion by loosely coupling my jobs? Projects planning to execute ensemble jobs to meet the INCITE criterion of utilizing a large fraction (e.g., 20%) of the leadership-class system for production simulations will be considered. The mission of the INCITE program is to enable high-impact, grand-challenge research that could not otherwise be performed without access to the leadership-class systems, Aurora and Frontier . Parameter sweeps, ensembles, design of experiments, and other statistical methods that require large numbers of discrete or loosely coupled simulations can be considered capability-class campaigns if the volume of work is so large that time-to-solution is an untenable pacing issue and if a software workflow solution (e.g., pre- and post-processing scripts that automate run management and analysis) is provided to facilitate this volume of work. Conversely, if by decoupling the simulations the work could be effectively carried out on a smaller resource within a reasonable time-to-solution, the project in all likelihood does not require the INCITE systems and will be assessed in this light. (A smaller resource is defined as a non-leadership-class system available through other allocation programs, e.g. institutional systems, nationally funded computing resources, etc.). Bundling together small jobs solely for the purpose of meeting the criterion for usage of a large fraction of LCF HPC will not be judged as capability computing. INCITE awards are made based on the potential for impact of the research and the need to carry out capability-class simulations.
What constitutes a Data and/or AI research project? Proposals in these areas should have strong aspects of scalable data processing (data-intensive computing, experimental/observational/simulation data analytics, etc.) and/or machine learning/AI (deep learning, neural networks, discovery of patterns and reduced models for scientific data and/or simulation modeling, etc.). As guidance on relevant research areas for proposals to this call, consider these suggestions: complex and interactive workflows, streaming/real-time data analysis, statistical methods, graph analytics, uncertainty quantification, deep learning, machine learning, hyper-parameter optimization, pattern recognition and classification, and machine learning integrated with applications and/or steering simulations. This guidance includes methodological advances in data science that require use of the LCF capabilities, and also applications that deploy these methods to advance their outcomes. Cross-cutting projects that integrate simulation and large-scale data science are welcome.
What does readiness mean for data-intensive/AI projects? See above for “What constitutes a “computationally intensive” research project?” Data analytics and/or AI proposals should demonstrate computational readiness by providing performance data (representative compute, memory, network, or disk usage) which support the required scale and/or the time demands of the problem and demonstrate the need for LCF resources. Performance data should demonstrate that the application is optimized for the resources requested (in terms of efficiency, scalability, throughput, data input/output, workflow tools for ensemble runs, checkpointing etc.). We encourage PIs planning to submit a data analytics and/or AI project to contact INCITE management and the LCFs if there are additional questions, in particular, regarding application software requirements. Because we anticipate that these projects may have unique requirements or challenges compared to traditional model-driven simulation campaigns, these projects are encouraged to collaborate closely with the LCF.
What software and technical infrastructure exist for data projects? OLCF and ALCF have infrastructure and software that is directly aimed at data science projects including workflow infrastructure (e.g., Slate at OLCF, Petrel at ALCF, etc.), analytics notebooks, data transfer nodes, etc. See OLCF and ALCF websites for specifics.
Do scaling data have to be from the proposed machine? No. Because the applicant may not have previously had access to the machine requested, he or she can provide scaling data from any machine available to him or her. However, scaling data from the proposed machine is highly recommended for demonstrating computational readiness. Upon request, the centers can provide short-term allocations for the purpose of preparing for an INCITE proposal.
Are INCITE awards for a period of only 1 year? INCITE awards can be from one to three years in length. Multiyear awards must submit annual renewal requests, which will undergo peer review. Requests for multiyear awards will be subject to the highest standards of excellence and are expected to yield truly extraordinary results.
What kind of protections do the INCITE centers offer for data? The INCITE centers have implemented cyber-security programs at a moderate baseline with compensatory controls in accordance with National Institute of Standards and Technology Special Publication 800-53. Project-specific security requirements can also be discussed and potentially addressed; the proposer should contact the center directly for details. Risk is currently mitigated using available enterprise infrastructure and network-based tools to provide perimeter protection and vulnerability resolution. System- and application-level security controls are implemented in accordance with industry and federal best practices. All non-temporary user data reside on centralized file and archival storage systems that are regularly backed up. These data are protected and segmented from other users through standard access controls. The HPC centers cannot guarantee zero risks to information stored on their systems. Requests for processing sensitive (e.g., proprietary, or export-controlled) data must therefore be clearly identified in the proposal. Information-sensitivity concerns apply to the intent of the research, storing and compiling of source code, loading and execution of application software, input data for the application software, output data generated by the application software, and data resulting from analysis of output data. If, for example, the application software has an Export Control Classification Number (ECCN), prospective users should state this fact in the proposal and provide the ECCN number. Prospective users should work with the center to identify appropriate levels of data protection. Greater levels of protection not typically offered may be provided at a cost borne by the project.
What are my responsibilities if I receive an award? The PI of each awarded project will be expected to sign terms-of-use agreements with the center and provide periodic reports (e.g., quarterly, end of year). Project participants will also sign terms-of-use agreements and be asked to respond to an annual user survey. All INCITE users are expected to acknowledge the center and the program in publications resulting from their award. The appropriate acknowledgement is listed below for INCITE and the respective center. INCITE/ALCF Acknowledgement: An award of computer time was provided by the INCITE program. This research used resources of the Argonne Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC02-06CH11357. INCITE/OLCF Acknowledgement: An award of computer time was provided by the INCITE program. This research also used resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725. INCITE/ALCF/OLCF Acknowledgement: An award of computer time was provided by the INCITE program. This research used resources of both the Argonne and Oak Ridge Leadership Computing Facilities, which are DOE Office of Science User Facilities supported under contracts DE-AC02-06CH11357 and DE-AC05-00OR22725 Non-compliance with the Terms of Use Agreement may result in the suspension of computer access.
