HTCondor

You can enable HTCondor on a CycleCloud cluster by modifying the run_list in the configuration section of your cluster definition. There are three basic components of an HTCondor cluster. The first is the central manager, which provides the scheduling and management daemons. The second component is one or more schedulers, from which jobs are submitted into the system. The final component is one or more execute nodes, which are the hosts that perform the computation. A simple HTCondor template might look like:

[cluster htcondor]

  [[node manager]]
  ImageName = cycle.image.centos7
  MachineType = Standard_A4 # 8 cores

      [[[configuration]]]
      run_list = role[central_manager]

  [[node scheduler]]
  ImageName = cycle.image.centos7
  MachineType = Standard_A4 # 8 cores

      [[[configuration]]]
      run_list = role[condor_scheduler_role],role[filer_role],role[scheduler]

  [[nodearray execute]]
  ImageName = cycle.image.centos7
  MachineType = Standard_A1 # 1 core
  Count = 1

      [[[configuration]]]
      run_list = role[usc_execute]

When you import and start a cluster with this definition in CycleCloud, you get a manager and a scheduler node, and one execute node. You can add execute nodes to the cluster by using the cyclecloud add_node command. To add 10 more execute nodes, use the following command:

cyclecloud add_node htcondor -t execute -c 10

HTCondor Autoscaling

CycleCloud supports autoscaling for HTCondor. The software monitors the status of your queue and turns on and off nodes as needed to complete the work in an optimal amount of time and cost. To enable autoscaling for HTCondor, add Autoscale=true to your cluster definition:

[cluster htcondor]
Autoscale = True

HTCondor Advanced Usage

If you know the average runtime of jobs, define average_runtime (in minutes) in your job. CycleCloud uses that value to start the minimum number of nodes. For example, if five 10-minute jobs are submitted and average_runtime is set to 10, CycleCloud starts only one node instead of five.

Autoscale Nodearray

By default, HTCondor requests cores from the nodearray called execute. If a job requires a different nodearray (for example, if certain jobs within a workflow have a high memory requirement), specify a slot_type attribute for the job. For example, adding +slot_type = "highmemory" causes HTCondor to request a node from the highmemory nodearray instead of execute (this setting currently requires htcondor.slot_type = "highmemory" to be set in the nodearray's [[[configuration]]] section). This setting doesn't affect how HTCondor schedules the jobs, so you might want to include the slot_type startd attribute in the job's requirements or rank expressions. For example: Requirements = target.slot_type = "highmemory".

Submitting Jobs to HTCondor

The most generic way to submit jobs to an HTCondor scheduler is the command (run from a scheduler node):

condor_submit my_job.submit

A sample submit file might look like this:

      Universe = vanilla
      Executable = do_science
      Arguments = -v --win-prize=true
      Output = log/$(Cluster).$(Process).out
      Error = log/$(Cluster).$(Process).err
      Should_transfer_files = if_needed
      When_to_transfer_output = On_exit
      +average_runtime = 10
      +slot_type = "highmemory"
      Queue

HTCondor Configuration Reference

The following HTCondor-specific configuration options customize functionality:

HTCondor Auto-Generated Configuration File

HTCondor has a large number of configuration settings, including user-defined attributes. CycleCloud offers the ability to create a custom configuration file using attributes defined in the cluster:

CycleCloud supports a standard set of autostop attributes across schedulers: