Creating New Mission Config Files¶

Required Reading¶

There are many niche topics important to understand how SPICE and kernels work. The important things to know:

What are kernels?
What are the different kernel types?
What are the different kernel qualities?

Here are some resources:

Intro to Kernels, includes descriptions of kernel types.
Useful glossary of terms
Kerneldbgen, used to create ISIS DB files, docs has useful descriptions of Kernel qualities.
NAIF required reading on SPICE.

NAIF does not provide adequate resources to programmatically get all required metadata. ISIS solves this problem by creating "DB Files", PVL files containing metadata for every kernel in the ISIS data area. These DB files are predominately curated in that either the Kernel files are handwritten or a script (generally called makedb, co-located with the kernels in the kernel directory) is handwritten to use kerneldbgen app to cache most of the kernel metadata.

Issues with ISIS's approach:

DB files are co-located with kernels and, therefore, it is difficult to release them separately from the kernels.
Kernels are tied to the ISIS data area and cannot be used directly on NAIF-distributed kernels.
As these DB files are in the ISIS data area, they cannot be versioned and controlled separately.
If you add kernels, you have to regenerate the kernel files.
ISIS's code for searching and exploiting Kernels is not modular.
Kernel search works on a per-image basis and cannot be used for arbitrary SPICE queries.

SpiceQL more-or-less rewrites much of ISIS's Kernel searching and exploitation code.

Distinctions in SpiceQL's approach:

DB files, here referred to as Mission config files, are located as JSON files that can be shipped separately from data installs.
Config files leverage common Kernel naming schemes so you can use both NAIF-installed kernels but also kernels from USGS Astro or any other source.
Config files are source-controlled.
If you add Kernels, you can easily modify them locally, and contribute your changes for the new Kernels.
Sugar Spice modularizes most of the Kernel code for search and I/O.
Config files can be used for more generalized kernel queries

Understanding how ISIS stores Kernel metadata¶

Structure of ISIS data area, ignoring non-kernel data:

    .
    └── $ISISDATA/
        └── <mission>/
            └── kernels/
                ├── <Binary Kernel Type>/
                │   ├── <kernel db files>
                │   ├── <Kernel makedb files>
                │   └── <kernel files>
                └── <Text Kernel Type>/
                    ├── <kernel db files> 
                    └── <kernel files>

Where <mission> is the name of any particular mission (MRO, Odyssey, Messenger, etc.), <Kernel Type> is with a slightly different structure if the kernel type is binary (CK, SPK, etc.) or text (IK, IAK, LSK, etc.), This section focuses on the Kernel "makedb" and "db" files.

Each kernel directory (ck, spk, ik, etc.) has a file called kernels.[0-9]{4}.db where [0-9]{4} is the version of the db file. As ISIS kernel metadata is not version controlled, it uses a proprietary versioning system where the 4 numbers after the first extension delimiter is a version number.

SPK Kernel DB file, $ISISDATA/messenger/kernels/spk/kernels.0184.db

    Object = SpacecraftPosition
    RunTime = 2015-09-14T13:41:46

    Group = Dependencies
        LeapsecondKernel = $base/kernels/lsk/naif0011.tls
    End_Group

    Group = Selection
        Time = ("2004 AUG 03 07:14:39.393205 TDB",
                "2015 APR 30 19:27:09.351488 TDB")
        File = $messenger/kernels/spk/msgr_20040803_20150430_od431sc_2.bsp
        Type = Reconstructed
    End_Group
    End_Object
    End

Some explanations:

Object = SpacecraftPositionspecifies this is SPK metadata.
Group = Dependencies specifies Kernels required to furnish these kernels and extract meaningful data, in this case, an LSK. Usually, this is an LSK or SCLK. Sometimes includes Kernels for other reference frames.
Group = Selection specifies the criteria for selecting a particular kernel, in this case, the start and stop time range, the path of the Kernel, and the Kernel quality.

For binary kernels, these DB files are generally not handwritten, instead, the higher level information is written into a script, usually named makedb with information on required time kernels, and regexes for different kernel qualities and the kernel type.

Makedb script, $ISISDATA/messenger/kernels/spk/makedb

  #  Generate kernel database from existing kernels
  kerneldbgen to='kernels.????.db' type=SPK \
              recondir='$messenger/kernels/spk' \
              reconfilter=msgr_20040803*.bsp \
              lsk='$base/kernels/lsk/naif????.tls'

Things to note:

type=SPK describes the Kernel type, this populates the Object name in the output PVL. In this case, Object = SpacecraftPosition
lsk='$base/kernels/lsk/naif????.tls describes the LSK dependencies, sometimes you also see SCLK=<path> in cases where a specific SCLK is required. Populates the dependencies group, in this case LeapsecondKernel = $base/kernels/lsk/naif0011.tls
reconfilter=msgr_20040803*.bsp describes the format (using Unix globbing wildcards) for this mission's reconstructed kernels for the given kernel type (SPK). Files matching these patterns are globbed, then:
1. kerneldb gen uses NAIFs cspice toolkit to compute the time ranges for the kernels.
2. Different filters. predictfilter matches are labeled as predicted, reconfilter matches are labeled as reconstructed, etc.
3. This process works very well as the names for mission kernels are very well structured.

Text Kernel file, $ISISDATA/mro/kernels/ik/kernels.0003.db

    Object = Instrument
      Group = Selection
        Match = ("Instrument","InstrumentId","Marci")
        File  = ("mro", "kernels/ik/mro_marci_v??.ti")
      EndGroup

      Group = Selection
        Match = ("Instrument","InstrumentId","HIRISE")
        File  = ("mro", "kernels/ik/mro_hirise_v??.ti")
      EndGroup

      Group = Selection
        Match = ("Instrument","InstrumentId","CRISM")
        File  = ("mro", "kernels/ik/mro_crism_v??.ti")
      EndGroup
    EndObject

Here, like with most IK db files, the IK selection is instrument-specific, so the Match key is used by ISIS to specify image label keywords that must match to select this kernel. These usually have no makedb and are handwritten as cspice is not needed to extract kernel times.

The last example, a more complex example where kernel selection requires multiple kernels selected from the same kernel type, is used in situations where a particular kernel has time-based dependencies.

In this example, we use Messenger's MDIS. MDIS was mounted on a gimbal, so to get correct pointing information for MDIS relative to the Messenger spacecraft, the following frame chain is required:

  Messenger Spacecraft -> Gimbal -> MDIS

Therefore, we need one CK kernel for the spacecraft, the gimbal, and then for MDIS. ISIS accomplishes this by using 3 different Kernel db files: messenger_kernels.[0-9]{4}.db, messenger_mdis_att.[0-9]{4}.db and mdis_kernels.[0-9]{4}.db generated with separate called to kerneldbgen. A handwritten configuration file, called kernels.[0-9]{4}.conf, is used to describe their relationships.

Conf file for Messenger, $ISISDATA/messenger/kernels/ck/kernels.0001.conf

  Object = Instrument

    Group = Selection
      Match = ("Instrument","InstrumentId","MDIS-WAC")
      File = ("messenger", "kernels/ck/mdis_kernels.????.db")
      File = ("messenger", "kernels/ck/messenger_mdis_att_kernels.????.db")
      File = ("messenger", "kernels/ck/messenger_kernels.????.db")
    End_Group

    Group = Selection
      Match = ("Instrument","InstrumentId","MDIS-NAC")
      File = ("messenger", "kernels/ck/mdis_kernels.????.db")
      File = ("messenger", "kernels/ck/messenger_mdis_att_kernels.????.db")
      File = ("messenger", "kernels/ck/messenger_kernels.????.db")
  End_Group

  End_Object
  End

makedb file for Messenger, $ISISDATA/messenger/kernels/ck/kernels.0001.conf

  #  Generate kernel database from existing kernels

  kerneldbgen to='messenger_kernels.????.db' type=CK \
              recondir='$messenger/kernels/ck' \
              reconfilter='msgr_????_v??.bc' \
              smitheddir='$messenger/kernels/ck' \
              smithedfilter='msgr_mdis_sc????_usgs_v?.bc' \
              sclk='$messenger/kernels/sclk/messenger_????.tsc' \
              lsk='$base/kernels/lsk/naif????.tls'

  kerneldbgen to='messenger_mdis_att_kernels.????.db' type=CK \
              recondir='$messenger/kernels/ck' \
              reconfilter='msgr_mdis_sc??????_??????_sub_v?.bc' \
              sclk='$messenger/kernels/sclk/messenger_????.tsc' \
              lsk='$base/kernels/lsk/naif????.tls'

  kerneldbgen to='mdis_kernels.????.db' type=CK \
              recondir='$messenger/kernels/ck' \
              reconfilter='msgr_mdis_gm??????_??????v?.bc' \
              sclk='$messenger/kernels/sclk/messenger_????.tsc' \
              lsk='$base/kernels/lsk/naif????.tls'

Here, you can the how dependencies on other CK kernels are expressed. As MDIS-WAC and MDIS-NAC have the same dependencies, the two groups are the same.

Everything here, except kernel start and stop times, requires a knowledgeable individual to manually label kernel qualities and dependencies. Times are cached by kerneldbgen.

Translating .db/makedb files to a single configuration file¶

Warning

As this library is a work in progress, much of this is subject to change. With so many missions with different requirements, this format may miss supporting specific requirements for some missions. If you have any suggested changes for this process, feel free to open an issue and start a discussion.

Example Configuration File:

Here, we'll run through an example config file, again for Messenger, encapsulating a lot of the information put into ISIS between the .db and makedb files.

Things to note:

The JSON format mimics ISIS's directory structure for kernels. With the tree:

    <mission>.json
     └── "<Frame>" 
          ├── "<Binary Kernel Type>:"
          │    └── "<Kernel Quality>:"
          │        └── <list of regexes>       
          ├── "<Text Kernel Type>:"
          │     └── <list of regexes> 
          └── "deps:"
               └── <list of JSON pointers>

Kernel formats now use EMCAscript (i.e. regular expressions).
No times are cached, these are simply computed at runtime.
To resolve frame chains and dependencies on other JSON keywords, the deps object can be any arbitrary pointer to another JSON block which this frame depends on for SCLKs, CKs, etc.
Instead of configs going into a black box that resolves per image kernels, running a query returns JSON matching this format.

Example output for querying all kernels for Messenger: https://gist.github.com/Kelvinrr/3e383d62eead58c016b7165acdf64d60

The Flowchart¶

Identify the mission, create a new JSON file called <mission>.json, and try to stick to NAIF abbreviations (see: https://naif.jpl.nasa.gov/pub/naif/pds/data/)
Do binary kernels first (CKs, SPKs, etc.):
- Look at the makedb file, translate the wildcards to EMCAscript, and place them under their respective quality keyword
- If the list of regexes (or any list) has only one element, feel free to write it as a string, not an array
- Check for a config file, make sure the dependencies on other time-dependent kernels are handled with the deps keyword.
When doing text kernels:
- Simply look at the kernel.[0-9]{4}.db file (specifically the newest one). Mimic the regexes into EMCAscript.
- The key is to make sure the right text kernel is going to the right instrument. For example, IKs are sometimes instrument-specific but also can be shared by all instruments on a spacecraft.
Test by creating a new gtest running utils::listMissionKernels and manually confirming that all kernels are accounted for. Look at FunctionalTestsSpiceQueries.cpp for some examples.
Add new config files to CMakeLists.txt's SPICEQL_CONFIG_FILES variable to make sure they get installed.