mirror of https://github.com/trailofbits/buttercup synced 2026-06-21 14:11:39 +00:00

Files

T

2over12 ea2fd093d5 Program model integrate (#13 )

* Add documentation

* Add janusgraph custom configuration. Simply graph creation.

* Add properties to edges. Index on uri property.

* reformat

* Create libpng graph in a more reasonable amount of time.

* Update README.md

* Wrap indexing into function

* updating...

* update proto

* fix command

* Update settings for serve and processing indexing

* Add dockerfile to docker container

* Add trigger script

* Fix error with loading graphml file

* reformat

* Remove resource constraints for program model

* Change permissions of temp dir

* Update readme

* Update dockerignore

---------

Co-authored-by: Evan Downing <2077950+evandowning@users.noreply.github.com>

2025-02-18 11:59:16 -05:00

5.8 KiB

Raw Blame History

Dev

Documentation to get started on developing Program Model APIs.

Requirements

n2-standard-4 instance.

Docker (WIP)

(type -p wget >/dev/null || (sudo apt update && sudo apt-get install wget -y)) \
	&& sudo mkdir -p -m 755 /etc/apt/keyrings \
        && out=$(mktemp) && wget -nv -O$out https://cli.github.com/packages/githubcli-archive-keyring.gpg \
        && cat $out | sudo tee /etc/apt/keyrings/githubcli-archive-keyring.gpg > /dev/null \
	&& sudo chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg \
	&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null \
	&& sudo apt update \
	&& sudo apt install gh -y
sudo apt update
sudo apt install gh

github.com -> settings -> developer settings -> personal access tokens -> tokens (classic) -> generate new token
select "repo" scope
select "read:org" scope

gh auth login
GitHub.com
SSH
No
Paste an authentication token

gh release download v0.0.2 -R github.com/trailofbits/aixcc-kythe
<authenticate URL in browser for the first time>
rm kythe-v0.0.67.tar.gz

sudo apt install just

cd afc-crs-trail-of-bits/
cp env.template .env

mkdir crs_scratch/
git clone --recursive git@github.com:aixcc-finals/example-libpng.git crs_scratch/libpng

just run-indexer

<todo>

docker compose down

Local

Setup

cd afc-crs-trail-of-bits/
cd program-model/
mkdir crs_scratch/

Download libpng

git clone git@github.com:aixcc-finals/example-libpng.git crs_scratch/libpng

Download Kythe

mkdir -p scripts/gzs/

gh release download v0.0.2 -R github.com/trailofbits/aixcc-kythe -D scripts/gzs/

mkdir crs_scratch/opt/

tar -zxf scripts/gzs/kythe-v0.0.67.tar.gz --directory crs_scratch/opt/

If no download available, build Kythe

Takes about 3 hours to build.

Install Bazel

Follow instructions. You will need to install a specific version of Bazel depending on what the build output below outputs.

Install Dependencies

sudo apt install flex bison asciidoc graphviz source-highlight clang

Download and build Kythe

From documentation

git clone git@github.com:trailofbits/aixcc-kythe.git crs_scratch/aixcc-kythe

cd crs_scratch/aixcc-kythe

git checkout a301676c20db9849a06878fa9cc017907eb64d72

bazel build //kythe/release

mkdir ../opt/

tar -zxf bazel-bin/kythe/release/kythe-v0.0.67.tar.gz --directory ../opt/

cd ../

Usage

Run indexer with Kythe

From documentation

sudo apt install cmake

cd crs_scratch/
export KYTHE_ROOT_DIRECTORY=`pwd`/libpng
export KYTHE_CORPUS="myrepo"
export KYTHE_OUTPUT_DIRECTORY=`pwd`/kythe_output
mkdir -p "$KYTHE_OUTPUT_DIRECTORY"
export CMAKE_ROOT_DIRECTORY=`pwd`/libpng

cd libpng/

../opt/kythe-v0.0.67/tools/runextractor cmake -extractor=../opt/kythe-v0.0.67/extractors/cxx_extractor -sourcedir=$CMAKE_ROOT_DIRECTORY

Merge kzip files

cd crs_scratch/

./opt/kythe-v0.0.67/tools/kzip merge --output $KYTHE_OUTPUT_DIRECTORY/merged.kzip $KYTHE_OUTPUT_DIRECTORY/*.kzip

./opt/kythe-v0.0.67/tools/kzip info --input $KYTHE_OUTPUT_DIRECTORY/merged.kzip | jq .

./opt/kythe-v0.0.67/tools/kzip view $KYTHE_OUTPUT_DIRECTORY/merged.kzip | jq .

Upload to JanusGraph

Extract graphml file from kzip contents.

cd program-model/src/buttercup

mkdir ../../data/

time ../../crs_scratch/opt/kythe-v0.0.67/indexers/cxx_indexer --ignore_unimplemented ../../crs_scratch/kythe_output/merged.kzip | uv run program_model/indexer/entries_into_graphml.py --output ../../data/graph.xml

Create network for JanusGraph to talk to Cassandra backend storage.

docker network create janusgraph-net

Start Cassandra container.

cd program-model/

./start_cql.sh

Start JanusGraph container with custom configuration file.

cd program-model/

./start_jg.sh

Run graph creation.

cd program-model/
docker ps | grep janusgraph

time docker run --rm --link <container_name>:janusgraph --network janusgraph-net -e GREMLIN_REMOTE_HOSTS=janusgraph -v `pwd`/load_data.groovy:/opt/janusgraph/load_data.groovy janusgraph/janusgraph -- ./bin/gremlin.sh -e load_data.groovy

Verify graph creation. From documentation.

docker run --rm --link <container_name>:janusgraph --network janusgraph-net -e GREMLIN_REMOTE_HOSTS=janusgraph -it janusgraph/janusgraph ./bin/gremlin.sh

gremlin> :remote connect tinkerpop.server conf/remote.yaml
gremlin> :remote console
gremlin> g.V().count()
gremlin> g.E().count()

Clear graph.

gremlin> g.V().drop().iterate()
gremlin> g.E().drop().iterate()

Contributing

Before committing code, run the following to ensure that the code is formatted correctly.

cd program-model/

just reformat

just lint

TODO

Use Indexes: org.janusgraph.graphdb.transaction.StandardJanusGraphTx$3.execute - Query requires iterating over all vertices [[~label = x]]. For better performance, use indexes
Consider bulk loading if needed
- Not required at the moment.
Consider loading data from a file.
Create script to verify creation of graph is accurate.
Reconcile the differences in the number of nodes and edges for libpng example.

5.8 KiB Raw Blame History

Dev