Ingesting Executables

If the BSim database has been tailored to ingest special executable categories, the extra metadata associated with an executable must be explicitly calculated and stored on the formal Ghidra program in order for BSim to see it. This is also accomplished with a Ghidra script passed to analyzeHeadless. BSim expects to find specific category values stored as Ghidra program options under Program.PROGRAM_INFO. A value can be set by passing the category name and the value itself as strings to the setString method on the Options object. The following example sets values for the category "WebResource".

@Override
public void run() throws Exception {
        Options pl = currentProgram.getOptions(Program.PROGRAM_INFO);
            String value = discoverWebResource();
            String value1 = discoverAnotherWebResource();
            Date compiledate = lookupCompileDate();
            pl.setString("WebResource",value);
            pl.setString("WebResource_1",value1);
            pl.setDate("Compile Date",compiledate);
}

The script has all the responsibility for constructing any actual value, and it can be run either as a pre-script or a post-script. Thus it can use the results of Ghidra's basic analysis to calculate the value or retrieve it from some external source like a CSV file. Once the script sets the value, assuming the executable category has already been added to the BSim database instance, there are no additional steps to take, and the rest of the BSim ingest process will make sure the value is incorporated into the database.

If more than one value needs to be assigned to the same category, the script must assign the first value to the option matching the category name and then assign additional options with names obtained by appending '_1', '_2', and so on to the category name. For the executable time-stamp, BSim reads the option matching the name of the specific Date executable category. This can be set within a script by using the setDate method instead of setString. If no Date category has been created, BSim will read the 'Date Created' option, which is normally filled in with the time at which Ghidra was created and analysis started. For additional discussion see “Executable Categories”.

If a set of function tags have been registered with the BSim database, some form of analysis must still place the tags on individual functions in a Ghidra program. As with executable categories, this can be accomplished with a Ghidra script passed to analyzeHeadless. Once a set of functions is identified, the tag(s) can be added using methods on the standard Function object. This script snippet looks up the function object by address and then sets the LOGGING tag on it and removes the SERIALPORT tag.

public void adjustTags(Address myaddress) throws Exception {
        Function func = program.getFunctionManager().getFunctionAt(myaddress);
        func.addTag("LOGGING");
        func.removeTag("SERIALPORT");
}

If the tag did not exist before the first call to addTag, it will be created. Assuming the tag has been registered with BSim, it will now automatically be included as part of the metadata of any functions it was added to. For additional discussion see “Function Tags”.

The database currently cannot remove individual function records. The records for an entire executable, and all the functions associated with it, can be removed. To remove an executable, use one of the following forms of the delete command:

In the --md5 form, you specify the 32 character hex representation of the md5 hash of the executable, which should identify it uniquely. Using the --name form, there is the possibility that the name is not unique, in which case the command will fail.

If a unique executable is identified, its metadata record will be removed, and the records for all functions which that executable formally contains will also be removed. For deployments which also maintain a loosely coupled Ghidra Server, keep in mind that removing executables from the BSim database with these commands does not remove the corresponding program from the Ghidra Server, this requires an additional step.

It is feasible to delete an executable from the database and then reingest it in order to populate updated information into the database, but this is fairly inefficient because you need to reperform all the function decompilation. Additionally this causes a large rearrangement of the database tables in order to perform what may amount to a very small change. The BSim database supports an update operation designed to make in-place changes to the metadata describing executables and functions. This is primarily useful for synchronizing function names with the BSim database as analysts rename them and commit their changes to a Ghidra Server, however other metadata, like executable architecture or category properties, can also be changed.

Very similar to the two ingest commands, generatesigs and commitsigs, there are also two update commands generateupdates and commitupdates which are invoked as follows:

The generateupdates command produces stripped down metadata XML files for every executable contained within the repository folder specified by the ghidraURL. Just like the generatesigs command, it can take an optional --config <config_template> parameter, which allows the command to execute without the BSim server running, otherwise a --bsim <bsimURL> parameter is required. It can also take an optional --overwrite parameter, causing it to overwrite any previously generated XML files. If the --bsim option is specified with the --commit option updates will be committed directly to the database. A BSim database commit is always performed using the specified bsimURL if an xmldirectory is not specified.

The commitupdates command commits the generated metadata to the BSim server. Only the smallest required change is issued as a formal transaction to the database, and the number of changes that are actually made are reported per executable. Functions and executables cannot be added or deleted to the BSim database using this command. If the update file describes new functions in a preexisting executable, this command will issue warnings about the existence of the new functions but will not create function records. Other functions will still get updated.

NOTE: Applies to PostgreSQL or Elasticsearch databases only

For those users performing large ingests or who find themselves rebuilding the database frequently, it is possible to drop the main index, ingest data, then recreate the main index. This is likely to be faster overall than the default behavior of updating the index as data is ingested. To drop the index, use the command:

Once the data has been ingested, rebuild the index with the command:

The time it takes to rebuild depends directly on the number of functions that have been ingested. For very large collections, rebuilding can take hours or days. The database can still be accessed while the index is dropped, but queries may take much longer to complete.

NOTE: Applies to PostgreSQL databases only

A maintainer can issue the bsim prewarm command to prepopulate RAM with commonly accessed portions of a BSim database.

Without this command, initial queries to a cold database (one that has just been restarted) can run slowly, giving poor response times until the cache is fully populated. A query typically needs to access an effectively random subset of pages making it a bad method for refreshing the cache.

The command is intended to be run once, immediately after restarting a server and before any queries have been made. It attempts to quickly preload the main vector index, and possibly portions of other tables, into RAM, by reading from disk sequentially. Queries may continue to improve as the database optimizes its cache across all tables, but the command effectively eliminates slow initial queries.