mirror of
https://github.com/mandiant/GoReSym
synced 2026-06-08 15:40:15 +00:00
Add -strings flag to extract Go strings from binaries (#77)
* Add -strings flag and ExtractMetadata field - Add Strings []string to ExtractMetadata struct - Add -strings command-line flag for string extraction - Update main_impl and main_impl_tmpfile signatures to accept printStrings parameter - Add placeholder string extraction logic with TODO marker - Update printForHuman to display extracted strings section - Verified flag appears in help and outputs correctly in both JSON and human format Part of #45 * Implement Go string extraction algorithm - Create objfile/strings.go with core extraction logic - Implement FLOSS-based string internment table detection - Add string candidate scanning (pointer + length pairs) - Implement findLongestMonotonicRun() for pattern detection - Add UTF-8 validation and printability filtering - Minimum string length: 4 characters, 80% printable - Add helper methods to elfFile: getSections(), is64Bit(), isLittleEndian() - Update main.go to call file.ExtractStrings() instead of placeholder - Tested with testproject/testproject: extracts 512 strings successfully - Extracts real Go strings: type names, runtime symbols, error messages Based on FLOSS floss/language/go/extract.py algorithm Part of #45 * Update README with -strings flag documentation - Add -strings flag to available flags list - Add Strings field to example JSON output - Document purpose: extract embedded Go strings from binary Part of #45 * Add tests validated against FLOSS output Per maintainer request, added comprehensive test suite: - strings_floss_test.go: Validates GoReSym against FLOSS reference output * 99.2% match rate (648/653 strings match FLOSS) * Uses FLOSS output from testproject.exe as ground truth * Reference saved in testdata/floss_reference.txt - strings_test.go: Additional unit tests for: * ELF and PE binary string extraction * Monotonic run detection algorithm * String filtering (printability, minimum length) - pe.go: Added helper methods (getSections, is64Bit, isLittleEndian) to enable string extraction from PE binaries All tests pass. * Address PR #77 review comments - Convert getSections() to iterateSections() using callback pattern to avoid memory pressure - Add Strings field to GoReSym.proto for external parsers - Implement iterateSections() for Mach-O format (previously missing) Changes requested by @stevemk14ebr in review: 1. Memory optimization: Replace array-based section loading with generator pattern 2. Proto definition: Add 'repeated string strings = 13' field 3. Mach-O support: Add missing iterateSections() implementation * Fix test failures and expand string extraction testing per PR #77 feedback - Fixed 4 test compilation errors by adding missing printStrings parameter to main_impl() calls - Added comprehensive TestStringExtraction function with 7 test cases covering Linux/macOS/Windows binaries - Implemented isPrintable() helper for ASCII validation (range 32-126) * Align string extraction with FLOSS algorithm per PR #77 review Rewrite string extraction to match FLOSS (extract.py) logic: - Sort candidates by address (not length) to fix monotonic run detection - Add image VA range and max section size filtering for candidates - Use candidate (pointer, length) pairs for direct extraction from blob - Replace 80% printable threshold with 100% fully printable check - Fix PE section addresses to include ImageBase for correct VA comparison Results: 648 strings extracted from PE test binary, 100% match with FLOSS. * Run string extraction on test build files * Extend main test to cover all new versions * Fix string extraction for old Windows binaries and add length sanity check 1. Add .text to data sections for old Go Windows binaries (1.7-1.10) that store strings in the code section instead of .rdata 2. Add maxReasonableStringLength (64KB) to filter out garbage candidates with huge lengths that cause incorrect blob boundary detection 3. Update TestIsDataSection to reflect .text being a valid data section Fixes string extraction failures on: - Old Windows PE binaries (Go 1.7-1.10): 0 -> 256 strings - Modern Windows: 574 strings - macOS: 284 strings All tests pass including TestWeirdBins on 15 real binaries. * Fix test failures for Go 1.5/1.6 and Go 1.22 macOS; fix build script typo 1. main_test.go: Skip Strings check for Go 1.5/1.6 Pre-SSA C-based linker does not produce length-sorted string blobs, so findLongestMonotonicRun never reaches the minimum threshold of 10. Pattern mirrors the existing interface-parsing guard. 2. objfile/strings.go: Handle prevNull == -1 in findStringBlobRange Apple's linker on newer Go/macOS (1.22+) packs sections without leading padding, so no null bytes exist before the first string candidate. bytes.LastIndex returns -1 in that case; treat it as offset 0 instead of bailing out with nil. 3. objfile/macho.go: Add missing is64Bit/isLittleEndian for machoFile elfFile and peFile already implement these interface methods; machoFile was the only rawFile implementation without them. Detect CPU type (CpuAmd64/Arm64 = 64-bit) and byte order from the macho.File struct. 4. build_test_files.sh: Fix $ver typo -> $GO_VER on mkdir line The directory was never created before Docker ran, causing Go 1.5 builds to produce no output. Now all 12 binaries are built for every version including 1.5 and 1.6. --------- Co-authored-by: Stephen Eckels <stevemk14ebr@gmail.com>
This commit is contained in:
kamran ul haq
@@ -74,4 +74,5 @@ message ExtractMetadata {
|
||||
repeated string files = 10 [json_name="Files"];
|
||||
repeated FuncMetadata userFunctions = 11 [json_name="UserFunctions"];
|
||||
repeated FuncMetadata stdFunctions = 12 [json_name="StdFunctions"];
|
||||
repeated string strings = 13 [json_name="Strings"];
|
||||
}
|
||||
|
||||
@@ -63,6 +63,7 @@ In this example, we ask GoReSym to recover type names (`-t`), user package names
|
||||
},
|
||||
"Types": [ ... ],
|
||||
"Files": [ ... ],
|
||||
"Strings": [ ... ],
|
||||
"UserFunctions": [ ... ],
|
||||
"StdFunctions": [ ... ]
|
||||
}
|
||||
@@ -73,6 +74,7 @@ Here are all the available flags:
|
||||
* `-d` ("default", optional) flag will print standard Go packages in addition to user packages.
|
||||
* `-p` ("paths", optional) flag will print any file paths embedded in the `pclntab`.
|
||||
* `-t` ("types", optional) flag will print Go type names.
|
||||
* `-strings` (optional) flag will extract embedded Go strings from the binary by analyzing the string internment table.
|
||||
* `-m <virtual address>` ("manual", optional) flag will dump the `RTYPE` structure recursively at the given virtual address
|
||||
* `-v <version string>` ("version", optional) flag will override automated version detection and use the provided version. This is needed for some stripped binaries. Type parsing will fail if the version is not accurate.
|
||||
* `-human` (optional) flag will print a flat text listing instead of JSON. Especially useful when printing structure and interface types.
|
||||
|
||||
+1
-1
@@ -6,7 +6,7 @@ for v in "${versions[@]}"
|
||||
do
|
||||
GO_TAG=$v
|
||||
GO_VER=$(echo "$GO_TAG" | tr -d '.')
|
||||
mkdir -p $(pwd)/test/build/"$ver"/
|
||||
mkdir -p $(pwd)/test/build/"$GO_VER"/
|
||||
|
||||
rm -f Dockerfile.test
|
||||
cat <<EOF >Dockerfile.test
|
||||
|
||||
@@ -64,9 +64,10 @@ type ExtractMetadata struct {
|
||||
Files []string
|
||||
UserFunctions []FuncMetadata
|
||||
StdFunctions []FuncMetadata
|
||||
Strings []string
|
||||
}
|
||||
|
||||
func main_impl_tmpfile(fileBytes []byte, printStdPkgs bool, printFilePaths bool, printTypes bool, noPrintFunctions bool, manualTypeAddress int, versionOverride string) (metadata ExtractMetadata, err error) {
|
||||
func main_impl_tmpfile(fileBytes []byte, printStdPkgs bool, printFilePaths bool, printTypes bool, noPrintFunctions bool, manualTypeAddress int, versionOverride string, printStrings bool) (metadata ExtractMetadata, err error) {
|
||||
tmpFile, err := os.CreateTemp(os.TempDir(), "goresym_tmp-")
|
||||
if err != nil {
|
||||
return ExtractMetadata{}, fmt.Errorf("failed to create temporary file: %s", err)
|
||||
@@ -81,10 +82,10 @@ func main_impl_tmpfile(fileBytes []byte, printStdPkgs bool, printFilePaths bool,
|
||||
return ExtractMetadata{}, fmt.Errorf("failed to close temporary file: %s", err)
|
||||
}
|
||||
|
||||
return main_impl(tmpFile.Name(), printStdPkgs, printFilePaths, printTypes, noPrintFunctions, manualTypeAddress, versionOverride)
|
||||
return main_impl(tmpFile.Name(), printStdPkgs, printFilePaths, printTypes, noPrintFunctions, manualTypeAddress, versionOverride, printStrings)
|
||||
}
|
||||
|
||||
func main_impl(fileName string, printStdPkgs bool, printFilePaths bool, printTypes bool, noPrintFunctions bool, manualTypeAddress int, versionOverride string) (metadata ExtractMetadata, err error) {
|
||||
func main_impl(fileName string, printStdPkgs bool, printFilePaths bool, printTypes bool, noPrintFunctions bool, manualTypeAddress int, versionOverride string, printStrings bool) (metadata ExtractMetadata, err error) {
|
||||
extractMetadata := ExtractMetadata{}
|
||||
|
||||
file, err := objfile.Open(fileName)
|
||||
@@ -287,6 +288,14 @@ restartParseWithRealTextBase:
|
||||
}
|
||||
}
|
||||
|
||||
if printStrings {
|
||||
strings, err := file.ExtractStrings()
|
||||
if err == nil {
|
||||
extractMetadata.Strings = strings
|
||||
}
|
||||
// If error, Strings will remain empty (nil slice)
|
||||
}
|
||||
|
||||
if !noPrintFunctions {
|
||||
for _, elem := range finalTab.ParsedPclntab.Funcs {
|
||||
if isStdPackage(elem.PackageName()) {
|
||||
@@ -376,6 +385,15 @@ func printForHuman(metadata ExtractMetadata) {
|
||||
fmt.Println("<NO FILES EXTRACTED>")
|
||||
}
|
||||
|
||||
fmt.Println("\n-Strings-")
|
||||
if len(metadata.Strings) > 0 {
|
||||
for _, str := range metadata.Strings {
|
||||
fmt.Println(str)
|
||||
}
|
||||
} else {
|
||||
fmt.Println("<NO STRINGS EXTRACTED>")
|
||||
}
|
||||
|
||||
fmt.Println("\n-User Functions-")
|
||||
if len(metadata.UserFunctions) > 0 {
|
||||
for i, fn := range metadata.UserFunctions {
|
||||
@@ -431,6 +449,7 @@ func main() {
|
||||
typeAddress := flag.Int("m", 0, "Manually parse the RTYPE at the provided virtual address, disables automated enumeration of moduledata typelinks itablinks")
|
||||
versionOverride := flag.String("v", "", "Override the automated version detection, ex: 1.17. If this is wrong, parsing may fail or produce nonsense")
|
||||
humanView := flag.Bool("human", false, "Human view, print information flat rather than json, some information is omitted for clarity")
|
||||
printStrings := flag.Bool("strings", false, "Extract embedded Go strings from binary")
|
||||
flag.Parse()
|
||||
|
||||
if *about {
|
||||
@@ -449,7 +468,7 @@ func main() {
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
metadata, err := main_impl(flag.Arg(0), *printStdPkgs, *printFilePaths, *printTypes, *noPrintFunctions, *typeAddress, *versionOverride)
|
||||
metadata, err := main_impl(flag.Arg(0), *printStdPkgs, *printFilePaths, *printTypes, *noPrintFunctions, *typeAddress, *versionOverride, *printStrings)
|
||||
if err != nil {
|
||||
fmt.Println(TextToJson("error", fmt.Sprintf("Failed to parse file: %s", err)))
|
||||
os.Exit(1)
|
||||
|
||||
+110
-5
@@ -11,7 +11,7 @@ import (
|
||||
_ "net/http/pprof"
|
||||
)
|
||||
|
||||
var versions = []string{"117", "116", "115", "114", "113", "112", "111", "110", "19", "18", "17", "16", "15"}
|
||||
var versions = []string{"122", "121", "120", "119", "118", "117", "116", "115", "114", "113", "112", "111", "110", "19", "18", "17", "16", "15"}
|
||||
var fileNames = []string{"testproject_lin", "testproject_lin_32", "testproject_lin_stripped", "testproject_lin_stripped_32", "testproject_mac", "testproject_mac_stripped", "testproject_win_32.exe", "testproject_win_stripped_32.exe", "testproject_win_stripped.exe", "testproject_win.exe"}
|
||||
|
||||
func TestAllVersions(t *testing.T) {
|
||||
@@ -30,7 +30,7 @@ func TestAllVersions(t *testing.T) {
|
||||
}
|
||||
|
||||
t.Run(versionPath, func(t *testing.T) {
|
||||
data, err := main_impl(filePath, true, true, true, false, 0, "")
|
||||
data, err := main_impl(filePath, true, true, true, false, 0, "", true)
|
||||
if err != nil {
|
||||
t.Errorf("Go %s failed on %s: %s", v, file, err)
|
||||
}
|
||||
@@ -107,6 +107,14 @@ func TestAllVersions(t *testing.T) {
|
||||
if data.Arch == "" {
|
||||
t.Errorf("Go %s Arch failed on %s: %s", v, file, err)
|
||||
}
|
||||
|
||||
// String extraction requires Go 1.7+ (SSA-based linker).
|
||||
// Old Go 1.5/1.6 used C linker which doesn't produce sorted string blobs.
|
||||
if v != "15" && v != "16" {
|
||||
if len(data.Strings) == 0 {
|
||||
t.Errorf("Go %s Strings failed on %s: %s", v, file, err)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -119,7 +127,7 @@ func testSymbolRecovery(t *testing.T, workingDirectory string, binaryName string
|
||||
return
|
||||
}
|
||||
|
||||
data, err := main_impl(filePath, true, true, true, false, 0, "")
|
||||
data, err := main_impl(filePath, true, true, true, false, 0, "", false)
|
||||
if err != nil {
|
||||
t.Errorf("GoReSym failed: %s", err)
|
||||
}
|
||||
@@ -214,7 +222,7 @@ func TestWeirdBins(t *testing.T) {
|
||||
return
|
||||
}
|
||||
|
||||
_, err := main_impl(filePath, true, true, true, false, 0, "")
|
||||
_, err := main_impl(filePath, true, true, true, false, 0, "", false)
|
||||
if err == nil {
|
||||
t.Errorf("GoReSym found pclntab in a non-go binary, this is not possible.")
|
||||
}
|
||||
@@ -228,7 +236,7 @@ func TestWeirdBins(t *testing.T) {
|
||||
return
|
||||
}
|
||||
|
||||
_, err := main_impl(filePath, true, true, true, false, 0, "")
|
||||
_, err := main_impl(filePath, true, true, true, false, 0, "", false)
|
||||
if err == nil {
|
||||
t.Errorf("GoReSym found pclntab in a non-go binary, this is not possible.")
|
||||
}
|
||||
@@ -245,3 +253,100 @@ func TestBig(t *testing.T) {
|
||||
testSymbolRecovery(t, workingDirectory, "kubectl_macho", 0x6C6CB20, 0x7F8CB20, 0x5CD9E40)
|
||||
})
|
||||
}
|
||||
|
||||
func isPrintable(s string) bool {
|
||||
if len(s) == 0 {
|
||||
return false
|
||||
}
|
||||
for _, r := range s {
|
||||
if r < 32 || r > 126 {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func TestStringExtraction(t *testing.T) {
|
||||
workingDirectory, err := os.Getwd()
|
||||
if err != nil {
|
||||
t.Errorf("Failed to get working directory")
|
||||
return
|
||||
}
|
||||
|
||||
// Test string extraction on multiple test binaries
|
||||
testCases := []struct {
|
||||
version string
|
||||
filename string
|
||||
minStrings int
|
||||
description string
|
||||
}{
|
||||
{"117", "testproject_lin", 100, "Go 1.17 Linux binary"},
|
||||
{"117", "testproject_lin_stripped", 100, "Go 1.17 Linux stripped binary"},
|
||||
{"117", "testproject_mac", 100, "Go 1.17 macOS binary"},
|
||||
{"117", "testproject_win.exe", 100, "Go 1.17 Windows binary"},
|
||||
{"116", "testproject_lin", 100, "Go 1.16 Linux binary"},
|
||||
{"115", "testproject_lin_32", 50, "Go 1.15 Linux 32-bit binary"},
|
||||
{"18", "testproject_mac_stripped", 100, "Go 1.8 macOS stripped binary"},
|
||||
}
|
||||
|
||||
for _, tc := range testCases {
|
||||
t.Run(fmt.Sprintf("%s_%s", tc.version, tc.filename), func(t *testing.T) {
|
||||
filePath := fmt.Sprintf("%s/test/build/%s/%s", workingDirectory, tc.version, tc.filename)
|
||||
if _, err := os.Stat(filePath); errors.Is(err, os.ErrNotExist) {
|
||||
t.Skipf("Test file %s doesn't exist", filePath)
|
||||
return
|
||||
}
|
||||
|
||||
// Extract with strings enabled
|
||||
data, err := main_impl(filePath, false, false, false, true, 0, "", true)
|
||||
if err != nil {
|
||||
t.Errorf("Failed to extract from %s: %v", tc.description, err)
|
||||
return
|
||||
}
|
||||
|
||||
// Check that strings were extracted
|
||||
if len(data.Strings) == 0 {
|
||||
t.Errorf("No strings extracted from %s", tc.description)
|
||||
return
|
||||
}
|
||||
|
||||
// Check minimum number of strings
|
||||
if len(data.Strings) < tc.minStrings {
|
||||
t.Errorf("Expected at least %d strings from %s, got %d", tc.minStrings, tc.description, len(data.Strings))
|
||||
}
|
||||
|
||||
// Check that all extracted strings are printable
|
||||
nonPrintableCount := 0
|
||||
for _, str := range data.Strings {
|
||||
if !isPrintable(str) {
|
||||
nonPrintableCount++
|
||||
}
|
||||
}
|
||||
|
||||
// Allow up to 5% non-printable strings (some edge cases may exist)
|
||||
maxNonPrintable := len(data.Strings) / 20
|
||||
if nonPrintableCount > maxNonPrintable {
|
||||
t.Errorf("Too many non-printable strings in %s: %d out of %d (max allowed: %d)",
|
||||
tc.description, nonPrintableCount, len(data.Strings), maxNonPrintable)
|
||||
}
|
||||
|
||||
// Check for some expected common Go strings
|
||||
expectedSubstrings := []string{"main", "go", "runtime"}
|
||||
foundCount := 0
|
||||
for _, expected := range expectedSubstrings {
|
||||
for _, str := range data.Strings {
|
||||
if strings.Contains(str, expected) {
|
||||
foundCount++
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if foundCount == 0 {
|
||||
t.Errorf("No common Go strings found in %s (expected at least one of: %v)", tc.description, expectedSubstrings)
|
||||
}
|
||||
|
||||
t.Logf("%s: extracted %d strings (%d printable)", tc.description, len(data.Strings), len(data.Strings)-nonPrintableCount)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -425,3 +425,34 @@ func (f *elfFile) loadAddress() (uint64, error) {
|
||||
func (f *elfFile) dwarf() (*dwarf.Data, error) {
|
||||
return f.elf.DWARF()
|
||||
}
|
||||
|
||||
// iterateSections calls the provided function for each section.
|
||||
// This avoids loading all section data into memory at once.
|
||||
func (f *elfFile) iterateSections(fn func(Section) error) error {
|
||||
for _, sec := range f.elf.Sections {
|
||||
data, err := sec.Data()
|
||||
if err != nil {
|
||||
// Skip sections we can't read
|
||||
continue
|
||||
}
|
||||
section := Section{
|
||||
Name: sec.Name,
|
||||
Addr: sec.Addr,
|
||||
Data: data,
|
||||
}
|
||||
if err := fn(section); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// is64Bit returns true if this is a 64-bit ELF file
|
||||
func (f *elfFile) is64Bit() bool {
|
||||
return f.elf.Class == elf.ELFCLASS64
|
||||
}
|
||||
|
||||
// isLittleEndian returns true if this is a little-endian ELF file
|
||||
func (f *elfFile) isLittleEndian() bool {
|
||||
return f.elf.Data == elf.ELFDATA2LSB
|
||||
}
|
||||
|
||||
@@ -419,3 +419,42 @@ func (f *machoFile) loadAddress() (uint64, error) {
|
||||
func (f *machoFile) dwarf() (*dwarf.Data, error) {
|
||||
return f.macho.DWARF()
|
||||
}
|
||||
|
||||
// iterateSections calls the provided function for each section.
|
||||
// This avoids loading all section data into memory at once.
|
||||
func (f *machoFile) iterateSections(fn func(Section) error) error {
|
||||
for _, sec := range f.macho.Sections {
|
||||
data, err := sec.Data()
|
||||
if err != nil {
|
||||
// Skip sections we can't read
|
||||
continue
|
||||
}
|
||||
section := Section{
|
||||
Name: sec.Name,
|
||||
Addr: sec.Addr,
|
||||
Data: data,
|
||||
}
|
||||
if err := fn(section); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// is64Bit returns true if this is a 64-bit Mach-O binary.
|
||||
// Go's debug/macho package represents CPU type directly on the File struct.
|
||||
func (f *machoFile) is64Bit() bool {
|
||||
switch f.macho.Cpu {
|
||||
case macho.CpuAmd64, macho.CpuArm64:
|
||||
return true
|
||||
default:
|
||||
return false // CpuI386, CpuArm, CpuPpc = 32-bit
|
||||
}
|
||||
}
|
||||
|
||||
// isLittleEndian returns true if this Mach-O binary is little-endian.
|
||||
// Modern Macs (x86-64, ARM64) are little-endian.
|
||||
// Old PowerPC Macs were big-endian.
|
||||
func (f *machoFile) isLittleEndian() bool {
|
||||
return f.macho.ByteOrder == binary.LittleEndian
|
||||
}
|
||||
|
||||
@@ -521,3 +521,50 @@ func (f *peFile) loadAddress() (uint64, error) {
|
||||
func (f *peFile) dwarf() (*dwarf.Data, error) {
|
||||
return f.pe.DWARF()
|
||||
}
|
||||
|
||||
// iterateSections calls the provided function for each section.
|
||||
// This avoids loading all section data into memory at once.
|
||||
// Section addresses are full VAs (ImageBase + VirtualAddress) so that
|
||||
// pointer values found in the binary data can be compared directly.
|
||||
func (f *peFile) iterateSections(fn func(Section) error) error {
|
||||
var imageBase uint64
|
||||
switch oh := f.pe.OptionalHeader.(type) {
|
||||
case *pe.OptionalHeader32:
|
||||
imageBase = uint64(oh.ImageBase)
|
||||
case *pe.OptionalHeader64:
|
||||
imageBase = oh.ImageBase
|
||||
}
|
||||
|
||||
for _, sec := range f.pe.Sections {
|
||||
data, err := sec.Data()
|
||||
if err != nil {
|
||||
// Skip sections we can't read
|
||||
continue
|
||||
}
|
||||
section := Section{
|
||||
Name: sec.Name,
|
||||
Addr: imageBase + uint64(sec.VirtualAddress),
|
||||
Data: data,
|
||||
}
|
||||
if err := fn(section); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// is64Bit returns true if this is a 64-bit PE file
|
||||
func (f *peFile) is64Bit() bool {
|
||||
switch f.pe.OptionalHeader.(type) {
|
||||
case *pe.OptionalHeader64:
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// isLittleEndian returns true if this is a little-endian PE file
|
||||
// PE files are always little-endian on x86/x64/ARM architectures
|
||||
func (f *peFile) isLittleEndian() bool {
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -0,0 +1,512 @@
|
||||
/*Copyright (C) 2022 Mandiant, Inc. All Rights Reserved.*/
|
||||
|
||||
// String extraction for Go binaries
|
||||
// Algorithm aligned with FLOSS (floss/language/go/extract.py)
|
||||
//
|
||||
// FLOSS reference: https://github.com/mandiant/flare-floss
|
||||
// Specifically: floss/language/go/extract.py
|
||||
//
|
||||
// The Go compiler stores strings in a "string blob" within data sections,
|
||||
// sorted by length (shortest to longest). Each string is referenced by a
|
||||
// struct { pointer, length } pair elsewhere in the binary.
|
||||
//
|
||||
// This algorithm:
|
||||
// 1. Scans data sections for candidate string structures (pointer + length pairs)
|
||||
// 2. Sorts candidates by pointer address (where the string data lives)
|
||||
// 3. Finds the longest monotonically increasing run of lengths, which
|
||||
// identifies candidates pointing into the string blob
|
||||
// 4. Locates the blob boundaries by searching for null terminator sequences
|
||||
// 5. Walks the blob using sorted pointers to extract individual strings
|
||||
//
|
||||
// Stack strings are out of scope (acceptable difference from FLOSS).
|
||||
|
||||
package objfile
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/binary"
|
||||
"fmt"
|
||||
"sort"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
const minStringLength = 4
|
||||
const maxReasonableStringLength = 65536 // 64KB - no real Go string should exceed this
|
||||
|
||||
// StringCandidate represents a potential Go string structure found in the binary.
|
||||
// Go strings are represented as a struct with a pointer and length:
|
||||
//
|
||||
// type string struct {
|
||||
// str unsafe.Pointer
|
||||
// len int
|
||||
// }
|
||||
type StringCandidate struct {
|
||||
Pointer uint64 // VA where the actual string data lives
|
||||
Length uint64 // Length of the string in bytes
|
||||
}
|
||||
|
||||
// ExtractStrings finds embedded Go strings in the binary by analyzing the
|
||||
// string internment table. Returns deduplicated, validated strings.
|
||||
func (f *File) ExtractStrings() ([]string, error) {
|
||||
var allStrings []string
|
||||
|
||||
for _, entry := range f.entries {
|
||||
strings, err := entry.extractStrings()
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
allStrings = append(allStrings, strings...)
|
||||
}
|
||||
|
||||
return allStrings, nil
|
||||
}
|
||||
|
||||
// extractStrings performs string extraction for a single Entry.
|
||||
//
|
||||
// This is the main orchestration function, following the FLOSS algorithm:
|
||||
//
|
||||
// FLOSS: get_string_blob_strings() in extract.py:266
|
||||
func (e *Entry) extractStrings() ([]string, error) {
|
||||
is64bit := e.is64Bit()
|
||||
isLittleEndian := e.isLittleEndian()
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Step 1a: Compute the binary's virtual address range
|
||||
// ---------------------------------------------------------------
|
||||
// FLOSS: low, high = get_image_range(pe) in utils.py:71
|
||||
//
|
||||
// Candidates whose pointer falls outside the binary's address space
|
||||
// are noise. We compute the range from ALL sections (not just data).
|
||||
var imageMin, imageMax uint64
|
||||
var maxSectionSize uint64
|
||||
first := true
|
||||
|
||||
err := e.iterateSections(func(section Section) error {
|
||||
// Skip sections with no virtual address (e.g., ELF .shstrtab,
|
||||
// .symtab, .strtab, debug sections, and the null section).
|
||||
// These are metadata sections not mapped to virtual memory.
|
||||
if section.Addr == 0 {
|
||||
return nil
|
||||
}
|
||||
sectionEnd := section.Addr + uint64(len(section.Data))
|
||||
if first || section.Addr < imageMin {
|
||||
imageMin = section.Addr
|
||||
}
|
||||
if first || sectionEnd > imageMax {
|
||||
imageMax = sectionEnd
|
||||
}
|
||||
if uint64(len(section.Data)) > maxSectionSize {
|
||||
maxSectionSize = uint64(len(section.Data))
|
||||
}
|
||||
first = false
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Step 1b: Collect string structure candidates from data sections
|
||||
// ---------------------------------------------------------------
|
||||
// We also keep section data for blob boundary detection later.
|
||||
// Only data sections are kept (not code), so memory impact is limited.
|
||||
var allCandidates []StringCandidate
|
||||
var dataSections []Section
|
||||
|
||||
err = e.iterateSections(func(section Section) error {
|
||||
if !isDataSection(section.Name) {
|
||||
return nil
|
||||
}
|
||||
dataSections = append(dataSections, section)
|
||||
candidates := findStringCandidates(section.Data, section.Addr, is64bit, isLittleEndian, imageMin, imageMax, maxSectionSize)
|
||||
allCandidates = append(allCandidates, candidates...)
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if len(allCandidates) == 0 {
|
||||
return []string{}, nil
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Step 2: Sort candidates by pointer address
|
||||
// ---------------------------------------------------------------
|
||||
// FLOSS: struct_strings.sort(key=lambda s: s.address)
|
||||
//
|
||||
// This is the KEY difference from the previous implementation which
|
||||
// sorted by length. Sorting by address groups candidates that point
|
||||
// into the same memory region (the string blob) together.
|
||||
sort.Slice(allCandidates, func(i, j int) bool {
|
||||
return allCandidates[i].Pointer < allCandidates[j].Pointer
|
||||
})
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Step 3: Find longest monotonically increasing run of lengths
|
||||
// ---------------------------------------------------------------
|
||||
// FLOSS: find_longest_monotonically_increasing_run(lengths)
|
||||
//
|
||||
// Since Go stores string data sorted by length in the blob, candidates
|
||||
// pointing into the blob will have monotonically increasing lengths
|
||||
// when sorted by address. This run is typically hundreds/thousands of
|
||||
// entries long, far longer than any random run.
|
||||
runStart, runEnd := findLongestMonotonicRun(allCandidates)
|
||||
if runStart == -1 || runEnd == -1 {
|
||||
return []string{}, nil
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Step 4: Find string blob boundaries
|
||||
// ---------------------------------------------------------------
|
||||
// FLOSS: find_string_blob_range() in extract.py:210
|
||||
//
|
||||
// Pick the mid-point candidate (to avoid edge corruption), find the
|
||||
// section containing its string data, then search for |00 00 00 00|
|
||||
// null sequences before and after to delimit the blob.
|
||||
blobStart, blobEnd, blobData := findStringBlobRange(allCandidates, runStart, runEnd, dataSections)
|
||||
if blobData == nil {
|
||||
return []string{}, nil
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------
|
||||
// Step 5: Extract strings using candidate (pointer, length) pairs
|
||||
// ---------------------------------------------------------------
|
||||
// FLOSS walks consecutive pointer pairs, augmented with LEA xrefs
|
||||
// (PE-specific, requires disassembly) for additional granularity.
|
||||
//
|
||||
// Since we operate cross-platform (ELF/PE/Mach-O), we instead use
|
||||
// each candidate's own (pointer, length) to extract its exact string
|
||||
// from the blob. This is more precise: each Go string struct already
|
||||
// knows its exact length, so we don't need pointer-gap inference.
|
||||
//
|
||||
// The blob boundary still serves its purpose: only candidates whose
|
||||
// pointer falls within the blob are considered (filtering noise).
|
||||
seen := make(map[string]bool)
|
||||
var result []string
|
||||
|
||||
for _, c := range allCandidates {
|
||||
// Only consider candidates pointing into the blob
|
||||
if c.Pointer < blobStart || c.Pointer >= blobEnd {
|
||||
continue
|
||||
}
|
||||
|
||||
offset := c.Pointer - blobStart
|
||||
if offset+c.Length > uint64(len(blobData)) {
|
||||
continue
|
||||
}
|
||||
|
||||
buf := blobData[offset : offset+c.Length]
|
||||
|
||||
// FLOSS: sbuf.decode("utf-8") -- skip on UnicodeDecodeError
|
||||
if !utf8.Valid(buf) {
|
||||
continue
|
||||
}
|
||||
|
||||
s := string(buf)
|
||||
|
||||
if len(s) < minStringLength {
|
||||
continue
|
||||
}
|
||||
|
||||
// Maintainer requirement: 100% printable (not 80%)
|
||||
if !isFullyPrintable(s) {
|
||||
continue
|
||||
}
|
||||
|
||||
// Deduplicate: multiple struct string candidates may reference
|
||||
// the same string (e.g., same string used in different packages)
|
||||
if seen[s] {
|
||||
continue
|
||||
}
|
||||
seen[s] = true
|
||||
|
||||
result = append(result, s)
|
||||
}
|
||||
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// iterateSections calls the provided function for each section in the binary.
|
||||
// Uses a callback pattern to avoid memory pressure from loading all sections.
|
||||
func (e *Entry) iterateSections(fn func(Section) error) error {
|
||||
if sectioner, ok := e.raw.(interface {
|
||||
iterateSections(func(Section) error) error
|
||||
}); ok {
|
||||
return sectioner.iterateSections(fn)
|
||||
}
|
||||
return fmt.Errorf("binary format does not support section enumeration")
|
||||
}
|
||||
|
||||
// Section represents a binary section with its virtual address and raw data.
|
||||
type Section struct {
|
||||
Name string
|
||||
Addr uint64
|
||||
Data []byte
|
||||
}
|
||||
|
||||
// isDataSection returns true if the section name suggests it contains data
|
||||
// (string structures or string blob data).
|
||||
func isDataSection(name string) bool {
|
||||
dataNames := []string{
|
||||
".rodata", ".data", ".noptrdata", // ELF
|
||||
"__rodata", "__data", "__noptrdata", // Mach-O
|
||||
".rdata", ".text", // PE (.text needed for old Go 1.7-1.10 Windows binaries)
|
||||
}
|
||||
for _, dataName := range dataNames {
|
||||
if name == dataName || name == dataName+".__" {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// findStringCandidates scans binary data for potential Go string structures.
|
||||
// Each candidate is a (pointer, length) pair found at pointer-aligned offsets.
|
||||
//
|
||||
// Filtering (aligned with FLOSS utils.py:331):
|
||||
// - Pointer must fall within the binary's VA range [imageMin, imageMax)
|
||||
// - Length must be > 0 and <= maxSectionSize
|
||||
// - Both pointer and length must be non-zero
|
||||
//
|
||||
// FLOSS equivalent: get_struct_string_candidates_with_pointer_size() in utils.py:331
|
||||
func findStringCandidates(data []byte, baseAddr uint64, is64bit bool, isLittleEndian bool, imageMin, imageMax, maxSectionSize uint64) []StringCandidate {
|
||||
var candidates []StringCandidate
|
||||
ptrSize := 4
|
||||
if is64bit {
|
||||
ptrSize = 8
|
||||
}
|
||||
|
||||
structSize := ptrSize * 2
|
||||
if len(data) < structSize {
|
||||
return candidates
|
||||
}
|
||||
|
||||
for i := 0; i <= len(data)-structSize; i += ptrSize {
|
||||
var ptr, length uint64
|
||||
|
||||
if is64bit {
|
||||
if isLittleEndian {
|
||||
ptr = binary.LittleEndian.Uint64(data[i : i+8])
|
||||
length = binary.LittleEndian.Uint64(data[i+8 : i+16])
|
||||
} else {
|
||||
ptr = binary.BigEndian.Uint64(data[i : i+8])
|
||||
length = binary.BigEndian.Uint64(data[i+8 : i+16])
|
||||
}
|
||||
} else {
|
||||
if isLittleEndian {
|
||||
ptr = uint64(binary.LittleEndian.Uint32(data[i : i+4]))
|
||||
length = uint64(binary.LittleEndian.Uint32(data[i+4 : i+8]))
|
||||
} else {
|
||||
ptr = uint64(binary.BigEndian.Uint32(data[i : i+4]))
|
||||
length = uint64(binary.BigEndian.Uint32(data[i+4 : i+8]))
|
||||
}
|
||||
}
|
||||
|
||||
// FLOSS: skips address==0, length==0
|
||||
if ptr == 0 || length == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
// FLOSS: if length > limit (max section size), skip
|
||||
if length > maxSectionSize {
|
||||
continue
|
||||
}
|
||||
|
||||
// Additional sanity check: reject unreasonably large strings
|
||||
// Real Go strings rarely exceed 64KB; larger values are likely garbage
|
||||
if length > maxReasonableStringLength {
|
||||
continue
|
||||
}
|
||||
|
||||
// FLOSS: if not (low <= address < high), skip
|
||||
// The pointer must point within the binary's virtual address space.
|
||||
if ptr < imageMin || ptr >= imageMax {
|
||||
continue
|
||||
}
|
||||
|
||||
candidates = append(candidates, StringCandidate{
|
||||
Pointer: ptr,
|
||||
Length: length,
|
||||
})
|
||||
}
|
||||
|
||||
return candidates
|
||||
}
|
||||
|
||||
// findLongestMonotonicRun finds the longest consecutive subsequence where each
|
||||
// candidate's Length is >= the previous candidate's Length.
|
||||
//
|
||||
// IMPORTANT: candidates must already be sorted by Pointer address before calling.
|
||||
// The lengths form a monotonically increasing pattern for candidates pointing
|
||||
// into the Go string blob (because Go stores strings sorted by length).
|
||||
//
|
||||
// FLOSS equivalent: find_longest_monotonically_increasing_run() in extract.py:143
|
||||
//
|
||||
// Example with address-sorted candidates:
|
||||
//
|
||||
// Pointer: 0x4000 0x4005 0x400A 0x4011 0x5000 0x5002
|
||||
// Length: 5 5 7 4 2 1
|
||||
// ^^^^^^^^^^^^^^^^^^^
|
||||
// monotonic run (5, 5, 7) = the string blob candidates
|
||||
// ^^^^^^^^^^^^^^^^^^^
|
||||
// non-blob (lengths decrease)
|
||||
func findLongestMonotonicRun(candidates []StringCandidate) (start, end int) {
|
||||
if len(candidates) == 0 {
|
||||
return -1, -1
|
||||
}
|
||||
|
||||
maxRunLength := 0
|
||||
maxRunEndIndex := 0
|
||||
|
||||
currentRunLength := 0
|
||||
var priorLength uint64
|
||||
|
||||
for i, c := range candidates {
|
||||
if c.Length >= priorLength {
|
||||
currentRunLength++
|
||||
} else {
|
||||
currentRunLength = 1
|
||||
}
|
||||
|
||||
if currentRunLength > maxRunLength {
|
||||
maxRunLength = currentRunLength
|
||||
maxRunEndIndex = i
|
||||
}
|
||||
|
||||
priorLength = c.Length
|
||||
}
|
||||
|
||||
maxRunStartIndex := maxRunEndIndex - maxRunLength + 1
|
||||
|
||||
// Real string tables have hundreds/thousands of entries
|
||||
if maxRunLength < 10 {
|
||||
return -1, -1
|
||||
}
|
||||
|
||||
return maxRunStartIndex, maxRunEndIndex
|
||||
}
|
||||
|
||||
// findStringBlobRange locates the string blob boundaries in memory.
|
||||
//
|
||||
// Algorithm (from FLOSS extract.py:210):
|
||||
// 1. Pick the mid-point candidate from the monotonic run (avoids edge junk)
|
||||
// 2. Find the section containing that candidate's string data
|
||||
// 3. Search forward for |00 00 00 00| to find blob end
|
||||
// 4. Search backward for |00 00 00 00| to find blob start
|
||||
//
|
||||
// Returns the blob's VA range and raw data, or nil if not found.
|
||||
func findStringBlobRange(candidates []StringCandidate, runStart, runEnd int, sections []Section) (blobStart, blobEnd uint64, blobData []byte) {
|
||||
// Pick mid-point to avoid junk at the edges
|
||||
// FLOSS: run_mid = (run_start + run_end) // 2
|
||||
runMid := (runStart + runEnd) / 2
|
||||
midCandidate := candidates[runMid]
|
||||
|
||||
// Find the section containing this string's data
|
||||
// FLOSS: section = pe.get_section_by_rva(instance_rva)
|
||||
var section *Section
|
||||
for i := range sections {
|
||||
sectionEnd := sections[i].Addr + uint64(len(sections[i].Data))
|
||||
if midCandidate.Pointer >= sections[i].Addr && midCandidate.Pointer < sectionEnd {
|
||||
section = §ions[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
if section == nil {
|
||||
return 0, 0, nil
|
||||
}
|
||||
|
||||
// Calculate offset within section
|
||||
// FLOSS: instance_offset = instance_rva - section.VirtualAddress
|
||||
instanceOffset := int(midCandidate.Pointer - section.Addr)
|
||||
if instanceOffset < 0 || instanceOffset >= len(section.Data) {
|
||||
return 0, 0, nil
|
||||
}
|
||||
|
||||
// Search for |00 00 00 00| boundaries
|
||||
// FLOSS uses this larger needle because some binaries have embedded |00 00|
|
||||
// See: https://github.com/Arker123/flare-floss/pull/3#issuecomment-1623354852
|
||||
nullNeedle := []byte{0x00, 0x00, 0x00, 0x00}
|
||||
|
||||
// Search forward from the instance for the blob end
|
||||
// FLOSS: next_null = section_data.find(b"\x00\x00\x00\x00", instance_offset)
|
||||
nextNullRel := bytes.Index(section.Data[instanceOffset:], nullNeedle)
|
||||
if nextNullRel == -1 {
|
||||
return 0, 0, nil
|
||||
}
|
||||
nextNull := instanceOffset + nextNullRel
|
||||
|
||||
// Search backward from the instance for the blob start
|
||||
// FLOSS: prev_null = section_data.rfind(b"\x00\x00\x00\x00", 0, instance_offset)
|
||||
prevNull := bytes.LastIndex(section.Data[:instanceOffset], nullNeedle)
|
||||
if prevNull == -1 {
|
||||
// String blob starts at the beginning of the section.
|
||||
// This happens in newer Go/macOS (Go 1.22+) where Apple's linker
|
||||
// packs sections without leading padding.
|
||||
prevNull = 0
|
||||
}
|
||||
|
||||
// Convert section-relative offsets to VAs
|
||||
// FLOSS: blob_start, blob_end = (section_start + prev_null, section_start + next_null)
|
||||
blobStart = section.Addr + uint64(prevNull)
|
||||
blobEnd = section.Addr + uint64(nextNull)
|
||||
blobData = section.Data[prevNull:nextNull]
|
||||
|
||||
return blobStart, blobEnd, blobData
|
||||
}
|
||||
|
||||
// isFullyPrintable returns true if ALL characters in the string are printable.
|
||||
// This replaces the previous isMostlyPrintable (80% threshold).
|
||||
//
|
||||
// Printable means: unicode.IsPrint(r) returns true, OR the character is a
|
||||
// common whitespace character (tab, newline, carriage return).
|
||||
//
|
||||
// Maintainer requirement: "ensure all strings are fully printable"
|
||||
func isFullyPrintable(s string) bool {
|
||||
if len(s) == 0 {
|
||||
return false
|
||||
}
|
||||
|
||||
for _, r := range s {
|
||||
if !unicode.IsPrint(r) && r != '\t' && r != '\n' && r != '\r' {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// deduplicateUint64 removes duplicate values from a sorted slice.
|
||||
func deduplicateUint64(sorted []uint64) []uint64 {
|
||||
if len(sorted) <= 1 {
|
||||
return sorted
|
||||
}
|
||||
result := make([]uint64, 0, len(sorted))
|
||||
result = append(result, sorted[0])
|
||||
for i := 1; i < len(sorted); i++ {
|
||||
if sorted[i] != sorted[i-1] {
|
||||
result = append(result, sorted[i])
|
||||
}
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// is64Bit determines if the binary is 64-bit
|
||||
func (e *Entry) is64Bit() bool {
|
||||
if bitChecker, ok := e.raw.(interface {
|
||||
is64Bit() bool
|
||||
}); ok {
|
||||
return bitChecker.is64Bit()
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// isLittleEndian determines if the binary is little-endian
|
||||
func (e *Entry) isLittleEndian() bool {
|
||||
if endianChecker, ok := e.raw.(interface {
|
||||
isLittleEndian() bool
|
||||
}); ok {
|
||||
return endianChecker.isLittleEndian()
|
||||
}
|
||||
return true
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
// Copyright (C) 2024 Mandiant, Inc. All Rights Reserved.
|
||||
package objfile
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestExtractStrings_CompareWithFLOSS validates our Go string extraction
|
||||
// against FLOSS's output. The reference output was generated using:
|
||||
// python -m floss.language.go.extract testproject/testproject.exe -n 4
|
||||
func TestExtractStrings_CompareWithFLOSS(t *testing.T) {
|
||||
// Load FLOSS reference output
|
||||
flossOutputPath := filepath.Join("..", "testdata", "floss_reference.txt")
|
||||
file, err := os.Open(flossOutputPath)
|
||||
if err != nil {
|
||||
t.Skipf("FLOSS reference output not found: %v", err)
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
// Parse FLOSS output (format: "0x12345: string content")
|
||||
flossStrings := make(map[string]bool)
|
||||
scanner := bufio.NewScanner(file)
|
||||
for scanner.Scan() {
|
||||
line := scanner.Text()
|
||||
if strings.HasPrefix(line, "0x") {
|
||||
parts := strings.SplitN(line, ": ", 2)
|
||||
if len(parts) == 2 {
|
||||
flossStrings[parts[1]] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
t.Logf("FLOSS reference contains %d strings", len(flossStrings))
|
||||
|
||||
// Extract strings using GoReSym
|
||||
testBinary := filepath.Join("..", "testproject", "testproject.exe")
|
||||
f, err := Open(testBinary)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to open test binary: %v", err)
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
goresymStrings, err := f.ExtractStrings()
|
||||
if err != nil {
|
||||
t.Fatalf("ExtractStrings() failed: %v", err)
|
||||
}
|
||||
|
||||
// Convert to set
|
||||
goresymSet := make(map[string]bool)
|
||||
for _, s := range goresymStrings {
|
||||
goresymSet[s] = true
|
||||
}
|
||||
|
||||
// Calculate overlap
|
||||
inBoth := 0
|
||||
for s := range goresymSet {
|
||||
if flossStrings[s] {
|
||||
inBoth++
|
||||
}
|
||||
}
|
||||
|
||||
goresymCount := len(goresymSet)
|
||||
matchRate := float64(inBoth) / float64(goresymCount) * 100
|
||||
|
||||
t.Logf("GoReSym extracted: %d strings", goresymCount)
|
||||
t.Logf("Overlap with FLOSS: %d strings (%.1f%% match rate)", inBoth, matchRate)
|
||||
|
||||
// Validate high match rate (>= 95%)
|
||||
if matchRate < 95.0 {
|
||||
t.Errorf("Match rate too low: %.1f%% (expected >= 95%%)", matchRate)
|
||||
|
||||
// Show examples of mismatches
|
||||
t.Log("Sample strings only in GoReSym:")
|
||||
count := 0
|
||||
for s := range goresymSet {
|
||||
if !flossStrings[s] && count < 5 {
|
||||
t.Logf(" %q", s)
|
||||
count++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Ensure reasonable extraction
|
||||
if goresymCount < 100 {
|
||||
t.Errorf("GoReSym extracted too few strings: %d (expected >= 100)", goresymCount)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,344 @@
|
||||
package objfile
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestExtractStrings_ELF tests string extraction from an ELF binary (Linux)
|
||||
func TestExtractStrings_ELF(t *testing.T) {
|
||||
testBinary := filepath.Join("..", "testproject", "testproject")
|
||||
|
||||
file, err := Open(testBinary)
|
||||
if err != nil {
|
||||
t.Skipf("Could not open test binary: %v (run: cd testproject && go build)", err)
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
strings, err := file.ExtractStrings()
|
||||
if err != nil {
|
||||
t.Fatalf("ExtractStrings() failed: %v", err)
|
||||
}
|
||||
|
||||
// Should extract a reasonable number of Go strings
|
||||
if len(strings) < 100 {
|
||||
t.Errorf("Expected at least 100 strings from ELF binary, got %d", len(strings))
|
||||
}
|
||||
|
||||
// Verify common Go type/keyword strings are present
|
||||
expectedStrings := []string{"func", "chan", "bool", "uint"}
|
||||
assertStringsPresent(t, strings, expectedStrings)
|
||||
}
|
||||
|
||||
// TestExtractStrings_PE tests string extraction from a PE binary (Windows)
|
||||
func TestExtractStrings_PE(t *testing.T) {
|
||||
testBinary := filepath.Join("..", "testproject", "testproject.exe")
|
||||
|
||||
file, err := Open(testBinary)
|
||||
if err != nil {
|
||||
t.Skipf("Could not open Windows test binary: %v (run: cd testproject && GOOS=windows GOARCH=amd64 go build -o testproject.exe)", err)
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
strings, err := file.ExtractStrings()
|
||||
if err != nil {
|
||||
t.Fatalf("ExtractStrings() failed: %v", err)
|
||||
}
|
||||
|
||||
// Should extract a reasonable number of Go strings
|
||||
if len(strings) < 100 {
|
||||
t.Errorf("Expected at least 100 strings from PE binary, got %d", len(strings))
|
||||
}
|
||||
|
||||
// Verify common Go type/keyword strings are present
|
||||
expectedStrings := []string{"func", "chan", "bool", "uint"}
|
||||
assertStringsPresent(t, strings, expectedStrings)
|
||||
}
|
||||
|
||||
// TestFindLongestMonotonicRun validates the core algorithm
|
||||
// Note: function requires at least 10 entries for a valid run (practical filter)
|
||||
func TestFindLongestMonotonicRun(t *testing.T) {
|
||||
t.Run("finds monotonic run in realistic data", func(t *testing.T) {
|
||||
// Create realistic test data: monotonically increasing lengths
|
||||
candidates := make([]StringCandidate, 50)
|
||||
for i := range candidates {
|
||||
candidates[i] = StringCandidate{
|
||||
Pointer: uint64(0x1000 + i*10),
|
||||
Length: uint64(i + 4), // lengths 4, 5, 6, ... 53
|
||||
}
|
||||
}
|
||||
|
||||
start, end := findLongestMonotonicRun(candidates)
|
||||
|
||||
// Should find the entire sequence as one run
|
||||
if start == -1 || end == -1 {
|
||||
t.Error("Expected to find a valid run, got (-1, -1)")
|
||||
}
|
||||
|
||||
runLength := end - start + 1
|
||||
if runLength < 10 {
|
||||
t.Errorf("Expected run length >= 10, got %d", runLength)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("too short returns -1", func(t *testing.T) {
|
||||
candidates := []StringCandidate{
|
||||
{Length: 1}, {Length: 2}, {Length: 3}, // Only 3 entries
|
||||
}
|
||||
start, end := findLongestMonotonicRun(candidates)
|
||||
if start != -1 || end != -1 {
|
||||
t.Errorf("Expected (-1, -1) for short input, got (%d, %d)", start, end)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestIsFullyPrintable validates the printability filter.
|
||||
// All characters must be printable (unicode.IsPrint) or common whitespace (\t, \n, \r).
|
||||
func TestIsFullyPrintable(t *testing.T) {
|
||||
tests := []struct {
|
||||
input string
|
||||
want bool
|
||||
}{
|
||||
{"hello world", true},
|
||||
{"runtime.error", true},
|
||||
{"line1\nline2", true}, // newlines are allowed
|
||||
{"col1\tcol2", true}, // tabs are allowed
|
||||
{"windows\r\n", true}, // carriage return allowed
|
||||
{"\x00\x01\x02\x03", false}, // all non-printable
|
||||
{"abc\x00\x01", false}, // any non-printable fails (was 80% threshold)
|
||||
{"mostly ok \x01", false}, // even one non-printable byte fails
|
||||
{"", false}, // empty string
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
got := isFullyPrintable(tt.input)
|
||||
if got != tt.want {
|
||||
t.Errorf("isFullyPrintable(%q) = %v, want %v", tt.input, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestExtractStrings_MinLength validates minimum length filtering
|
||||
func TestExtractStrings_MinLength(t *testing.T) {
|
||||
testBinary := filepath.Join("..", "testproject", "testproject")
|
||||
|
||||
file, err := Open(testBinary)
|
||||
if err != nil {
|
||||
t.Skip("Test binary not available")
|
||||
}
|
||||
defer file.Close()
|
||||
|
||||
strings, err := file.ExtractStrings()
|
||||
if err != nil {
|
||||
t.Fatalf("ExtractStrings() failed: %v", err)
|
||||
}
|
||||
|
||||
// All strings must be >= 4 characters (MIN_STRING_LENGTH)
|
||||
for _, s := range strings {
|
||||
if len(s) < 4 {
|
||||
t.Errorf("Found string shorter than 4 chars: %q (len=%d)", s, len(s))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestFindLongestMonotonicRun_MixedData tests that the algorithm correctly
|
||||
// finds the blob candidates among noise. This simulates address-sorted
|
||||
// candidates where only the middle portion points into the string blob.
|
||||
func TestFindLongestMonotonicRun_MixedData(t *testing.T) {
|
||||
// Simulate: 5 noise candidates, then 20 blob candidates, then 5 noise
|
||||
var candidates []StringCandidate
|
||||
|
||||
// Noise before blob (random lengths, not monotonic)
|
||||
noise1 := []uint64{50, 3, 100, 7, 42}
|
||||
for i, l := range noise1 {
|
||||
candidates = append(candidates, StringCandidate{
|
||||
Pointer: uint64(0x1000 + i*8),
|
||||
Length: l,
|
||||
})
|
||||
}
|
||||
|
||||
// String blob candidates: sorted by address, monotonically increasing lengths
|
||||
// This is what Go's string internment table looks like
|
||||
for i := 0; i < 20; i++ {
|
||||
candidates = append(candidates, StringCandidate{
|
||||
Pointer: uint64(0x4000 + i*10),
|
||||
Length: uint64(4 + i), // 4, 5, 6, ..., 23
|
||||
})
|
||||
}
|
||||
|
||||
// Noise after blob
|
||||
noise2 := []uint64{2, 80, 1, 60, 5}
|
||||
for i, l := range noise2 {
|
||||
candidates = append(candidates, StringCandidate{
|
||||
Pointer: uint64(0x8000 + i*8),
|
||||
Length: l,
|
||||
})
|
||||
}
|
||||
|
||||
start, end := findLongestMonotonicRun(candidates)
|
||||
|
||||
if start == -1 || end == -1 {
|
||||
t.Fatal("Expected to find a valid run, got (-1, -1)")
|
||||
}
|
||||
|
||||
runLength := end - start + 1
|
||||
if runLength != 20 {
|
||||
t.Errorf("Expected run length 20 (the blob candidates), got %d", runLength)
|
||||
}
|
||||
|
||||
// The run should start at index 5 (after the 5 noise entries)
|
||||
if start != 5 {
|
||||
t.Errorf("Expected run to start at index 5, got %d", start)
|
||||
}
|
||||
}
|
||||
|
||||
// TestDeduplicateUint64 validates the deduplication helper
|
||||
func TestDeduplicateUint64(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input []uint64
|
||||
expect []uint64
|
||||
}{
|
||||
{"no duplicates", []uint64{1, 2, 3}, []uint64{1, 2, 3}},
|
||||
{"with duplicates", []uint64{1, 1, 2, 3, 3, 3, 4}, []uint64{1, 2, 3, 4}},
|
||||
{"all same", []uint64{5, 5, 5}, []uint64{5}},
|
||||
{"single element", []uint64{42}, []uint64{42}},
|
||||
{"empty", []uint64{}, []uint64{}},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := deduplicateUint64(tt.input)
|
||||
if len(got) != len(tt.expect) {
|
||||
t.Errorf("deduplicateUint64(%v) returned %d elements, want %d", tt.input, len(got), len(tt.expect))
|
||||
return
|
||||
}
|
||||
for i := range got {
|
||||
if got[i] != tt.expect[i] {
|
||||
t.Errorf("deduplicateUint64(%v)[%d] = %d, want %d", tt.input, i, got[i], tt.expect[i])
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestIsDataSection validates section name matching across ELF/PE/Mach-O formats
|
||||
func TestIsDataSection(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
want bool
|
||||
}{
|
||||
// ELF sections
|
||||
{".rodata", true},
|
||||
{".data", true},
|
||||
{".noptrdata", true},
|
||||
// Mach-O sections
|
||||
{"__rodata", true},
|
||||
{"__data", true},
|
||||
{"__noptrdata", true},
|
||||
// PE sections
|
||||
{".rdata", true},
|
||||
{".text", true}, // Included for old Go Windows binaries (1.7-1.10) that store strings in .text
|
||||
// Suffixed variants (some formats append .__)
|
||||
{".rodata.__", true},
|
||||
// Non-data sections
|
||||
{".bss", false},
|
||||
{"__TEXT", false},
|
||||
{".got", false},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
got := isDataSection(tt.name)
|
||||
if got != tt.want {
|
||||
t.Errorf("isDataSection(%q) = %v, want %v", tt.name, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestFindStringCandidates validates candidate extraction from raw binary data
|
||||
func TestFindStringCandidates(t *testing.T) {
|
||||
t.Run("64-bit little-endian", func(t *testing.T) {
|
||||
// Create a fake section with one string struct:
|
||||
// struct: pointer=0x5000, length=5
|
||||
// Scanner steps at ptrSize (8-byte) alignment, so we use exactly
|
||||
// one 16-byte struct to avoid overlapping false positives.
|
||||
data := make([]byte, 16)
|
||||
// pointer = 0x5000 (little-endian uint64)
|
||||
data[0] = 0x00
|
||||
data[1] = 0x50
|
||||
data[2] = 0x00
|
||||
data[3] = 0x00
|
||||
data[4] = 0x00
|
||||
data[5] = 0x00
|
||||
data[6] = 0x00
|
||||
data[7] = 0x00
|
||||
// length = 5 (little-endian uint64)
|
||||
data[8] = 0x05
|
||||
data[9] = 0x00
|
||||
data[10] = 0x00
|
||||
data[11] = 0x00
|
||||
data[12] = 0x00
|
||||
data[13] = 0x00
|
||||
data[14] = 0x00
|
||||
data[15] = 0x00
|
||||
|
||||
// imageMin=0x1000, imageMax=0x10000 (pointer 0x5000 is in range), maxSectionSize=1000
|
||||
candidates := findStringCandidates(data, 0x1000, true, true, 0x1000, 0x10000, 1000)
|
||||
|
||||
if len(candidates) != 1 {
|
||||
t.Fatalf("Expected 1 candidate, got %d", len(candidates))
|
||||
}
|
||||
if candidates[0].Pointer != 0x5000 || candidates[0].Length != 5 {
|
||||
t.Errorf("candidate[0] = {%#x, %d}, want {0x5000, 5}", candidates[0].Pointer, candidates[0].Length)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("skips zero pointer and length", func(t *testing.T) {
|
||||
data := make([]byte, 16)
|
||||
// pointer=0, length=0 -- should be skipped
|
||||
candidates := findStringCandidates(data, 0x1000, true, true, 0x1000, 0x10000, 1000)
|
||||
if len(candidates) != 0 {
|
||||
t.Errorf("Expected 0 candidates for zero data, got %d", len(candidates))
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("skips pointer outside image range", func(t *testing.T) {
|
||||
data := make([]byte, 16)
|
||||
// pointer = 0x5000 (outside range [0x1000, 0x4000))
|
||||
data[0] = 0x00
|
||||
data[1] = 0x50
|
||||
data[8] = 0x05
|
||||
candidates := findStringCandidates(data, 0x1000, true, true, 0x1000, 0x4000, 1000)
|
||||
if len(candidates) != 0 {
|
||||
t.Errorf("Expected 0 candidates for out-of-range pointer, got %d", len(candidates))
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("skips length exceeding max section size", func(t *testing.T) {
|
||||
data := make([]byte, 16)
|
||||
// pointer = 0x2000 (in range), length = 500 (exceeds maxSectionSize=100)
|
||||
data[0] = 0x00
|
||||
data[1] = 0x20
|
||||
data[8] = 0xF4 // 500 in little-endian
|
||||
data[9] = 0x01
|
||||
candidates := findStringCandidates(data, 0x1000, true, true, 0x1000, 0x10000, 100)
|
||||
if len(candidates) != 0 {
|
||||
t.Errorf("Expected 0 candidates for oversized length, got %d", len(candidates))
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Helper function to check if expected strings are present
|
||||
func assertStringsPresent(t *testing.T, strings []string, expected []string) {
|
||||
t.Helper()
|
||||
stringSet := make(map[string]bool)
|
||||
for _, s := range strings {
|
||||
stringSet[s] = true
|
||||
}
|
||||
|
||||
for _, exp := range expected {
|
||||
if !stringSet[exp] {
|
||||
t.Errorf("Expected string %q not found in extracted strings", exp)
|
||||
}
|
||||
}
|
||||
}
|
||||
Vendored
+1750
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user