feat: add brute force search fallback for elf #89

Zxilly · 2024-02-13T20:55:40Z

Close: #79
Close: #76

I did some test on the binaries. Seems the fallback only need for elf files.
On macho files, __gopclntab always exists.

bin-darwin-1.11-amd64         : MACHO: __gopclntab
bin-darwin-1.11-amd64-ext     : MACHO: __gopclntab
bin-darwin-1.11-amd64-ext-pie : MACHO: __gopclntab
bin-darwin-1.11-amd64-pie     : MACHO: __gopclntab
bin-darwin-1.13-amd64         : MACHO: __gopclntab
bin-darwin-1.13-amd64-ext     : MACHO: __gopclntab
bin-darwin-1.13-amd64-ext-pie : MACHO: __gopclntab
bin-darwin-1.13-amd64-pie     : MACHO: __gopclntab
bin-darwin-1.15-amd64         : MACHO: __gopclntab
bin-darwin-1.15-amd64-ext     : MACHO: __gopclntab
bin-darwin-1.15-amd64-ext-pie : MACHO: __gopclntab
bin-darwin-1.15-amd64-pie     : MACHO: __gopclntab
bin-darwin-1.17-amd64         : MACHO: __gopclntab
bin-darwin-1.17-amd64-ext     : MACHO: __gopclntab
bin-darwin-1.17-amd64-ext-pie : MACHO: __gopclntab
bin-darwin-1.17-amd64-pie     : MACHO: __gopclntab
bin-darwin-1.19-amd64         : MACHO: __gopclntab
bin-darwin-1.19-amd64-ext     : MACHO: __gopclntab
bin-darwin-1.19-amd64-ext-pie : MACHO: __gopclntab
bin-darwin-1.19-amd64-pie     : MACHO: __gopclntab
bin-darwin-1.20-amd64-ext     : MACHO: __gopclntab
bin-darwin-1.21-amd64         : MACHO: __gopclntab
bin-darwin-1.21-amd64-ext     : MACHO: __gopclntab
bin-darwin-1.21-amd64-ext-pie : MACHO: __gopclntab
bin-linux-1.11-amd64          : ELF: .gopclntab
bin-linux-1.11-amd64-ext      : ELF: .gopclntab
bin-linux-1.11-amd64-ext-pie  : ELF: .data.rel.ro
bin-linux-1.11-amd64-pie      : ELF: .data.rel.ro
bin-linux-1.13-amd64          : ELF: .gopclntab
bin-linux-1.13-amd64-ext      : ELF: .gopclntab
bin-linux-1.13-amd64-ext-pie  : ELF: .data.rel.ro
bin-linux-1.13-amd64-pie      : ELF: .data.rel.ro
bin-linux-1.15-amd64          : ELF: .gopclntab
bin-linux-1.15-amd64-ext      : ELF: .gopclntab
bin-linux-1.15-amd64-ext-pie  : ELF: .data.rel.ro
bin-linux-1.15-amd64-pie      : ELF: .data.rel.ro.gopclntab
bin-linux-1.17-amd64          : ELF: .gopclntab
bin-linux-1.17-amd64-ext      : ELF: .gopclntab
bin-linux-1.17-amd64-ext-pie  : ELF: .data.rel.ro
bin-linux-1.17-amd64-pie      : ELF: .data.rel.ro.gopclntab
bin-linux-1.18-amd64          : ELF: .gopclntab
bin-linux-1.19-amd64          : ELF: .gopclntab
bin-linux-1.19-amd64-ext      : ELF: .gopclntab
bin-linux-1.19-amd64-ext-pie  : ELF: .data.rel.ro
bin-linux-1.19-amd64-pie      : ELF: .data.rel.ro.gopclntab
bin-linux-1.20-amd64-ext      : ELF: .gopclntab
bin-linux-1.21-amd64          : ELF: .gopclntab
bin-linux-1.21-amd64-ext      : ELF: .gopclntab
bin-linux-1.21-amd64-ext-pie  : ELF: .data.rel.ro
bin-windows-1.11-amd64        : PE: .rdata
bin-windows-1.11-amd64-ext    : PE: .rdata
bin-windows-1.13-amd64        : PE: .rdata
bin-windows-1.13-amd64-ext    : PE: .rdata
bin-windows-1.15-amd64        : PE: .rdata
bin-windows-1.15-amd64-ext    : PE: .rdata
bin-windows-1.15-amd64-ext-pie: PE: .rdata
bin-windows-1.15-amd64-pie    : PE: .rdata
bin-windows-1.17-amd64        : PE: .rdata
bin-windows-1.17-amd64-ext    : PE: .rdata
bin-windows-1.17-amd64-ext-pie: PE: .rdata
bin-windows-1.17-amd64-pie    : PE: .rdata
bin-windows-1.19-amd64        : PE: .rdata
bin-windows-1.19-amd64-ext    : PE: .rdata
bin-windows-1.19-amd64-ext-pie: PE: .rdata
bin-windows-1.19-amd64-pie    : PE: .rdata
bin-windows-1.20-amd64-ext    : PE: .rdata
bin-windows-1.21-amd64        : PE: .rdata
bin-windows-1.21-amd64-ext    : PE: .rdata
bin-windows-1.21-amd64-ext-pie: PE: .rdata

codecov-commenter · 2024-02-13T21:31:15Z

Codecov Report

Attention: Patch coverage is 63.72392% with 226 lines in your changes are missing coverage. Please review.

Project coverage is 76.07%. Comparing base (de41912) to head (6242d7b).
Report is 18 commits behind head on develop.

Files	Patch %	Lines
moduledata.go	46.71%	66 Missing and 15 partials ⚠️
elf.go	40.59%	56 Missing and 4 partials ⚠️
moduledata_gen.go	77.29%	47 Missing ⚠️
pe.go	76.47%	15 Missing and 5 partials ⚠️
macho.go	77.77%	7 Missing and 3 partials ⚠️
file.go	50.00%	4 Missing and 2 partials ⚠️
pclntab.go	87.50%	2 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop      #89      +/-   ##
===========================================
- Coverage    78.87%   76.07%   -2.80%     
===========================================
  Files           15       17       +2     
  Lines         3474     3983     +509     
===========================================
+ Hits          2740     3030     +290     
- Misses         518      712     +194     
- Partials       216      241      +25

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Zxilly · 2024-02-14T08:11:38Z

After #69 gets merged, tryFromSchedinit can be rewritten to based on symbol or dwarf.

Zxilly · 2024-02-14T18:44:55Z

@TcM1911 This patch adds some logic based on the symbol table. But most of the binary in the https://github.com/goretk/testdata are stripped and cannot be used to test.

I write a script to build different binaries for research at https://github.com/Zxilly/go-testdata/blob/master/scripts/build.py. Hope this can help testdata to provide more pre-compiled examples.

TcM1911 · 2024-02-19T09:46:21Z

We need a solution that works for binaries without symbols. To support stripped binaries, we essentially need to find and process the PCLNTAB and the moduledata structures at the same time. Finding the PCLNTAB first is easiest because it starts with a set of magic bytes. So here is how I suggest going about this:

Find where the PCLNTAB is located.
Use the address to search for the moduledata since the first entry is the PCLNTAB address.
Once we have a possible moduledata location, check the fields for start and end of the code section. From a fast look, it appears that we only have two possible offset values between all the different moduledata structure versions. The values read must be within the text segment so we check that against the segments from the file header.
Once we have the start of "Go code section" we can parse the PCLNTAB and use that information to find the Go version followed by parsing the moduledata.

Zxilly · 2024-02-19T22:04:05Z

Currently it can work without symtab. But with symtab it works much more faster.

Zxilly · 2024-02-19T22:08:31Z

One major problem is that without symbol tables and debugging information, the extration of golang version can only rely on regular matches. But according to my research on compilers, buildinfo is usually at the very beginning and thus false positives are unlikely.

We could merge this now and add a brute force pattern matching pclntab scheme for case when golang version info is not found.

Zxilly · 2024-02-22T11:17:03Z

I would like to know if any progress has been made and I will continue to complete the brute force search fallback on this PR.

Zxilly · 2024-02-22T21:01:34Z

brute force search fallback for no goversion case has been added. Hope someone can review this.

Zxilly · 2024-02-23T18:04:41Z

I think there must be a bug in the code path of the symbol process, but it's hard to find it because the binary provided didn't include the unstrip version.

Zxilly · 2024-03-07T06:57:44Z

@TcM1911 any updates?

TcM1911 · 2024-03-07T16:59:30Z

@Zxilly, sorry I've been very busy with other things. I actually got a chance to work on a solution to this myself today. I may have a chance to finish it tomorrow.

TcM1911 · 2024-03-08T12:28:27Z

@Zxilly, check out this PR: #90

Zxilly · 2024-03-08T18:18:53Z

Wait for #90 merged

Zxilly force-pushed the search branch from 21e84c7 to b554fa5 Compare February 13, 2024 21:16

Zxilly added 5 commits February 15, 2024 01:18

feat: add brute force search fallback for elf

bc71163

fix: disable tryFromSchedInit

45c48ab

fix: add missing filehandler

851482c

fix: add fallback section for pe

7ebb7e2

docs: update cycle dependency comment

4bdf611

Zxilly force-pushed the search branch from 8277322 to 4bdf611 Compare February 14, 2024 17:21

Zxilly added 2 commits February 15, 2024 01:22

fix: move to new api

9f3e2f5

feat: allow tryFromSchedInit works with symbol

7a7903f

Zxilly added 5 commits February 22, 2024 20:14

refactor: remove duplicate pclntab getter

7c1b23a

feat: add moduledata list for later search

8b62444

fix: fix moduledata generate

8799837

fix: export fields to work with reflect

f744e21

feat: add fallback for no version found case

029108d

Zxilly added 3 commits February 23, 2024 05:02

chore: remove debug statement

cbd9d5a

fix: set correct behavior for windows

70d70b8

perf: make build simultaneously

33028db

Zxilly added 3 commits February 24, 2024 02:08

fix: waitgroup race

983ee3b

feat: make tests simultaneously

d6e2909

fix: fix wrong t instance

92fd146

Merge branch 'goretk:develop' into search

6242d7b

TcM1911 mentioned this pull request Mar 8, 2024

Handle files when external linker was used. #90

Merged

Zxilly closed this Mar 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add brute force search fallback for elf #89

feat: add brute force search fallback for elf #89

Zxilly commented Feb 13, 2024 •

edited

Loading

codecov-commenter commented Feb 13, 2024 •

edited

Loading

Zxilly commented Feb 14, 2024

Zxilly commented Feb 14, 2024

TcM1911 commented Feb 19, 2024

Zxilly commented Feb 19, 2024 •

edited

Loading

Zxilly commented Feb 19, 2024 •

edited

Loading

Zxilly commented Feb 22, 2024

Zxilly commented Feb 22, 2024

Zxilly commented Feb 23, 2024

Zxilly commented Mar 7, 2024

TcM1911 commented Mar 7, 2024

TcM1911 commented Mar 8, 2024

Zxilly commented Mar 8, 2024

feat: add brute force search fallback for elf #89

feat: add brute force search fallback for elf #89

Conversation

Zxilly commented Feb 13, 2024 • edited Loading

codecov-commenter commented Feb 13, 2024 • edited Loading

Codecov Report

Zxilly commented Feb 14, 2024

Zxilly commented Feb 14, 2024

TcM1911 commented Feb 19, 2024

Zxilly commented Feb 19, 2024 • edited Loading

Zxilly commented Feb 19, 2024 • edited Loading

Zxilly commented Feb 22, 2024

Zxilly commented Feb 22, 2024

Zxilly commented Feb 23, 2024

Zxilly commented Mar 7, 2024

TcM1911 commented Mar 7, 2024

TcM1911 commented Mar 8, 2024

Zxilly commented Mar 8, 2024

Zxilly commented Feb 13, 2024 •

edited

Loading

codecov-commenter commented Feb 13, 2024 •

edited

Loading

Zxilly commented Feb 19, 2024 •

edited

Loading

Zxilly commented Feb 19, 2024 •

edited

Loading