STRLCPY/syft

■ ■ ■ ■ ■ ■

.golangci.yaml

		skipped 5 lines
6	6		# include:
7	7		# - EXC0002 # disable excluding of issues about comments from golint
8	8
9		-
10	9		linters:
11	10		# inverted configuration with `enable-all` and `disable` is not scalable during updates of golangci-lint
12	11		disable-all: true
		skipped 1 lines
14	13		- asciicheck
15	14		- bodyclose
16	15		- depguard
	16	+	- dogsled
17	17		- dupl
18	18		- errcheck
19		-	- errorlint
20	19		- exportloopref
21		-	- forcetypeassert
22	20		- funlen
23	21		- gocognit
24	22		- goconst
25	23		- gocritic
26	24		- gocyclo
27	25		- gofmt
28		-	- tparallel
29		-	- importas
	26	+	- goimports
	27	+	- goprintffuncname
30	28		- gosec
31	29		- gosimple
32	30		- govet
33	31		- ineffassign
34	32		- misspell
35		-	- nolintlint
	33	+	- nakedret
36	34		- revive
37	35		- staticcheck
38	36		- stylecheck
		skipped 2 lines
41	39		- unparam
42	40		- unused
43	41		- whitespace
	42	+
44	43		linters-settings:
45	44		funlen:
46	45		# Checks the number of lines in a function.
		skipped 10 lines
57	56		timeout: 10m
58	57
59	58		# do not enable...
60		-	# - dogsled # found to be to niche and ineffective
	59	+	# - deadcode # The owner seems to have abandoned the linter. Replaced by "unused".
61	60		# - goprintffuncname # does not catch all cases and there are exceptions
62	61		# - nakedret # does not catch all cases and should not fail a build
63	62		# - gochecknoglobals
		skipped 9 lines
73	72		# - lll # without a way to specify per-line exception cases, this is not usable
74	73		# - maligned # this is an excellent linter, but tricky to optimize and we are not sensitive to memory layout optimizations
75	74		# - nestif
76		-	# - prealloc # following this rule isn't consistently a good idea, as it sometimes forces unnecessary allocations that result in less idiomatic code
77		-	# - scopelint # deprecated
	75	+	# - nolintlint # as of go1.19 this conflicts with the behavior of gofmt, which is a deal-breaker (lint-fix will still fail when running lint)
	76	+	# - prealloc # following this rule isn't consistently a good idea, as it sometimes forces unnecessary allocations that result in less idiomatic code
	77	+	# - rowserrcheck # not in a repo with sql, so this is not useful
	78	+	# - scopelint # deprecated
	79	+	# - structcheck # The owner seems to have abandoned the linter. Replaced by "unused".
78	80		# - testpackage
79		-	# - wsl # this doens't have an auto-fixer yet and is pretty noisy (https://github.com/bombsimon/wsl/issues/90)
	81	+	# - varcheck # The owner seems to have abandoned the linter. Replaced by "unused".
	82	+	# - wsl # this doens't have an auto-fixer yet and is pretty noisy (https://github.com/bombsimon/wsl/issues/90)
80	83

■ ■ ■ ■ ■ ■

DEVELOPING.md

		skipped 117 lines
118	118
119	119		Catalogers are the way in which syft is able to identify and construct packages given some amount of source metadata.
120	120		For example, Syft can locate and process `package-lock.json` files when performing filesystem scans.
121		-	See: [how to specify file globs](https://github.com/anchore/syft/blob/main/syft/pkg/cataloger/javascript/cataloger.go#L16-L21)
122		-	and an implementation of the [package-lock.json parser](https://github.com/anchore/syft/blob/main/syft/pkg/cataloger/javascript/cataloger.go#L16-L21) fora quick review.
	121	+	See: [how to specify file globs](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/cataloger/javascript/cataloger.go#L16-L21)
	122	+	and an implementation of the [package-lock.json parser](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/cataloger/javascript/cataloger.go#L16-L21) fora quick review.
123	123
124	124		#### Building a new Cataloger
125	125
126		-	Catalogers must fulfill the interface [found here](https://github.com/anchore/syft/blob/main/syft/pkg/cataloger.go).
	126	+	Catalogers must fulfill the interface [found here](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/cataloger.go).
127	127		This means that when building a new cataloger, the new struct must implement both method signatures of `Catalog` and `Name`.
128	128
129		-	A top level view of the functions that construct all the catalogers can be found [here](https://github.com/anchore/syft/blob/main/syft/pkg/cataloger/cataloger.go).
	129	+	A top level view of the functions that construct all the catalogers can be found [here](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/cataloger/cataloger.go).
130	130		When an author has finished writing a new cataloger this is the spot to plug in the new catalog constructor.
131	131
132		-	For a top level view of how the catalogers are used see [this function](https://github.com/anchore/syft/blob/main/syft/pkg/cataloger/catalog.go#L41-L100) as a reference. It ranges over all catalogers passed as an argument and invokes the `Catalog` method:
	132	+	For a top level view of how the catalogers are used see [this function](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/cataloger/catalog.go#L41-L100) as a reference. It ranges over all catalogers passed as an argument and invokes the `Catalog` method:
133	133
134	134		Each cataloger has its own `Catalog` method, but this does not mean that they are all vastly different.
135		-	Take a look at the `apkdb` cataloger for alpine to see how it [constructs a generic.NewCataloger](https://github.com/anchore/syft/blob/main/syft/pkg/cataloger/apkdb/cataloger.go).
	135	+	Take a look at the `apkdb` cataloger for alpine to see how it [constructs a generic.NewCataloger](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/cataloger/apkdb/cataloger.go).
136	136
137	137		`generic.NewCataloger` is an abstraction syft uses to make writing common components easier. First, it takes the `catalogerName` to identify the cataloger.
138	138		On the other side of the call it uses two key pieces which inform the cataloger how to identify and return packages, the `globPatterns` and the `parseFunction`:
139	139		- The first piece is a `parseByGlob` matching pattern used to identify the files that contain the package metadata.
140		-	See [here for the APK example](https://github.com/anchore/syft/blob/main/syft/pkg/apk_metadata.go#L16-L41).
	140	+	See [here for the APK example](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/apk_metadata.go#L16-L41).
141	141		- The other is a `parseFunction` which informs the cataloger what to do when it has found one of the above matches files.
142		-	See this [link for an example](https://github.com/anchore/syft/blob/main/syft/pkg/cataloger/apkdb/parse_apk_db.go#L22-L102).
	142	+	See this [link for an example](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/cataloger/apkdb/parse_apk_db.go#L22-L102).
143	143
144	144		If you're unsure about using the `Generic Cataloger` and think the use case being filled requires something more custom
145	145		just file an issue or ask in our slack, and we'd be more than happy to help on the design.
146	146
147		-	Identified packages share a common struct so be sure that when the new cataloger is constructing a new package it is using the [`Package` struct](https://github.com/anchore/syft/blob/main/syft/pkg/package.go#L16-L31).
	147	+	Identified packages share a common struct so be sure that when the new cataloger is constructing a new package it is using the [`Package` struct](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/package.go#L16-L31).
148	148
149	149		Metadata Note: Identified packages are also assigned specific metadata that can be unique to their environment.
150		-	See [this folder](https://github.com/anchore/syft/tree/main/syft/pkg) for examples of the different metadata types.
	150	+	See [this folder](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg) for examples of the different metadata types.
151	151		These are plugged into the `MetadataType` and `Metadata` fields in the above struct. `MetadataType` informs which type is being used. `Metadata` is an interface converted to that type.
152	152
153	153		Finally, here is an example of where the package construction is done in the apk cataloger. The first link is where `newPackage` is called in the `parseFunction`. The second link shows the package construction:
154		-	- [Call for new package](https://github.com/anchore/syft/blob/6a7d6e6071829c7ce2943266c0e187b27c0b325c/syft/pkg/cataloger/apkdb/parse_apk_db.go#L96-L99)
155		-	- [APK Package Constructor](https://github.com/anchore/syft/blob/6a7d6e6071829c7ce2943266c0e187b27c0b325c/syft/pkg/cataloger/apkdb/package.go#L12-L27)
	154	+	- [Call for new package](https://github.com/anchore/syft/blob/v0.70.0/syft/pkg/cataloger/apkdb/parse_apk_db.go#L106)
	155	+	- [APK Package Constructor](https://github.com/anchore/syft/tree/v0.70.0/syft/pkg/cataloger/apkdb/package.go#L12-L27)
156	156
157	157		If you have more questions about implementing a cataloger or questions about one you might be currently working
158	158		always feel free to file an issue or reach out to us [on slack](https://anchore.com/slack).
	159	+
	160	+	#### Searching for files
	161	+
	162	+	All catalogers are provided an instance of the [`source.FileResolver`](https://github.com/anchore/syft/blob/v0.70.0/syft/source/file_resolver.go#L8) to interface with the image and search for files. The implementations for these
	163	+	abstractions leverage [`stereoscope`](https://github.com/anchore/stereoscope) in order to perform searching. Here is a
	164	+	rough outline how that works:
	165	+
	166	+	1. a stereoscope `file.Index` is searched based on the input given (a path, glob, or MIME type). The index is relatively fast to search, but requires results to be filtered down to the files that exist in the specific layer(s) of interest. This is done automatically by the `filetree.Searcher` abstraction. This abstraction will fallback to searching directly against the raw `filetree.FileTree` if the index does not contain the file(s) of interest. Note: the `filetree.Searcher` is used by the `source.FileResolver` abstraction.
	167	+	2. Once the set of files are returned from the `filetree.Searcher` the results are filtered down further to return the most unique file results. For example, you may have requested for files by a glob that returns multiple results. These results are filtered down to deduplicate by real files, so if a result contains two references to the same file, say one accessed via symlink and one accessed via the real path, then the real path reference is returned and the symlink reference is filtered out. If both were accessed by symlink then the first (by lexical order) is returned. This is done automatically by the `source.FileResolver` abstraction.
	168	+	3. By the time results reach the `pkg.Cataloger` you are guaranteed to have a set of unique files that exist in the layer(s) of interest (relative to what the resolver supports).
159	169
160	170		## Testing
161	171
		skipped 154 lines

■ ■ ■ ■ ■ ■

go.mod

		skipped 51 lines
52	52		github.com/CycloneDX/cyclonedx-go v0.7.1-0.20221222100750-41a1ac565cce
53	53		github.com/Masterminds/sprig/v3 v3.2.3
54	54		github.com/anchore/go-logger v0.0.0-20220728155337-03b66a5207d8
55		-	github.com/anchore/stereoscope v0.0.0-20230203152723-c49244e4d66f
	55	+	github.com/anchore/stereoscope v0.0.0-20230208154630-5a306f07f2e7
56	56		github.com/docker/docker v23.0.0+incompatible
57	57		github.com/google/go-containerregistry v0.13.0
58	58		github.com/invopop/jsonschema v0.7.0
		skipped 1 lines
60	60		github.com/opencontainers/go-digest v1.0.0
61	61		github.com/sassoftware/go-rpmutils v0.2.0
62	62		github.com/vbatts/go-mtree v0.5.2
63		-	golang.org/x/exp v0.0.0-20220823124025-807a23277127
	63	+	golang.org/x/exp v0.0.0-20230202163644-54bba9f4231b
64	64		gopkg.in/yaml.v3 v3.0.1
65	65		)
66	66
		skipped 3 lines
70	70		github.com/Masterminds/semver/v3 v3.2.0 // indirect
71	71		github.com/Microsoft/go-winio v0.6.0 // indirect
72	72		github.com/anchore/go-struct-converter v0.0.0-20221118182256-c68fdcfa2092 // indirect
	73	+	github.com/becheran/wildmatch-go v1.0.0 // indirect
73	74		github.com/containerd/containerd v1.6.12 // indirect
74	75		github.com/containerd/stargz-snapshotter/estargz v0.12.1 // indirect
75	76		github.com/davecgh/go-spew v1.1.1 // indirect
		skipped 51 lines
127	128		golang.org/x/sync v0.1.0 // indirect
128	129		golang.org/x/sys v0.5.0 // indirect
129	130		golang.org/x/text v0.7.0 // indirect
130		-	golang.org/x/tools v0.1.12 // indirect
	131	+	golang.org/x/tools v0.2.0 // indirect
131	132		golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
132	133		google.golang.org/genproto v0.0.0-20221227171554-f9683d7f8bef // indirect
133	134		google.golang.org/grpc v1.52.0 // indirect
		skipped 27 lines

■ ■ ■ ■ ■ ■

go.sum

		skipped 89 lines
90	90		github.com/anchore/go-version v1.2.2-0.20200701162849-18adb9c92b9b/go.mod h1:Bkc+JYWjMCF8OyZ340IMSIi2Ebf3uwByOk6ho4wne1E=
91	91		github.com/anchore/packageurl-go v0.1.1-0.20230104203445-02e0a6721501 h1:AV7qjwMcM4r8wFhJq3jLRztew3ywIyPTRapl2T1s9o8=
92	92		github.com/anchore/packageurl-go v0.1.1-0.20230104203445-02e0a6721501/go.mod h1:Blo6OgJNiYF41ufcgHKkbCKF2MDOMlrqhXv/ij6ocR4=
93		-	github.com/anchore/stereoscope v0.0.0-20230203152723-c49244e4d66f h1:hyEFgDzqZRr/+q1cPfjgIKXWJ7lMHDHmDXAOrhKMhRA=
94		-	github.com/anchore/stereoscope v0.0.0-20230203152723-c49244e4d66f/go.mod h1:YerDPu5voTWZUmjrAHhak7gGGdGLilqroEEFLA/PUHo=
	93	+	github.com/anchore/stereoscope v0.0.0-20230208154630-5a306f07f2e7 h1:PrdFBPMyika+AM1/AwDmYqrVeUATDU90wbrd81ugicU=
	94	+	github.com/anchore/stereoscope v0.0.0-20230208154630-5a306f07f2e7/go.mod h1:TUCfo52tEz7ahTUFtKN//wcB7kJzQs0Oifmnd4NkIXw=
95	95		github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883/go.mod h1:rCTlJbsFo29Kk6CurOXKm700vrz8f0KW0JNfpkRJY/8=
96	96		github.com/andybalholm/brotli v1.0.1/go.mod h1:loMXtMfwqflxFJPmdbJO0a3KNoPuLBgiu3qAvBg8x/Y=
97	97		github.com/andybalholm/brotli v1.0.4 h1:V7DdXeJtZscaqfNuAdSRuRFzuiKlHSC/Zh3zl9qY3JY=
		skipped 4 lines
102	102		github.com/armon/go-metrics v0.3.10/go.mod h1:4O98XIr/9W0sxpJ8UaYkvjk10Iff7SnFrb4QAOwNTFc=
103	103		github.com/armon/go-radix v0.0.0-20180808171621-7fddfc383310/go.mod h1:ufUuZ+zHj4x4TnLV4JWEpy2hxWSpsRywHrMgIH9cCH8=
104	104		github.com/armon/go-radix v1.0.0/go.mod h1:ufUuZ+zHj4x4TnLV4JWEpy2hxWSpsRywHrMgIH9cCH8=
	105	+	github.com/becheran/wildmatch-go v1.0.0 h1:mE3dGGkTmpKtT4Z+88t8RStG40yN9T+kFEGj2PZFSzA=
	106	+	github.com/becheran/wildmatch-go v1.0.0/go.mod h1:gbMvj0NtVdJ15Mg/mH9uxk2R1QCistMyU7d9KFzroX4=
105	107		github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
106	108		github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
107	109		github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
		skipped 529 lines
637	639		golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
638	640		golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM=
639	641		golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU=
640		-	golang.org/x/exp v0.0.0-20220823124025-807a23277127 h1:S4NrSKDfihhl3+4jSTgwoIevKxX9p7Iv9x++OEIptDo=
641		-	golang.org/x/exp v0.0.0-20220823124025-807a23277127/go.mod h1:cyybsKvd6eL0RnXn6p/Grxp8F5bW7iYuBgsNCOHpMYE=
	642	+	golang.org/x/exp v0.0.0-20230202163644-54bba9f4231b h1:EqBVA+nNsObCwQoBEHy4wLU0pi7i8a4AL3pbItPdPkE=
	643	+	golang.org/x/exp v0.0.0-20230202163644-54bba9f4231b/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc=
642	644		golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
643	645		golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
644	646		golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
		skipped 259 lines
904	906		golang.org/x/tools v0.1.3/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
905	907		golang.org/x/tools v0.1.4/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
906	908		golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
907		-	golang.org/x/tools v0.1.12 h1:VveCTK38A2rkS8ZqFY25HIDFscX5X9OoEhJd3quQmXU=
908	909		golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
	910	+	golang.org/x/tools v0.2.0 h1:G6AHpWxTMGY1KyEYoAQ5WTtIekUUvDNjan3ugu60JvE=
	911	+	golang.org/x/tools v0.2.0/go.mod h1:y4OqIKeOV/fWJetJ8bXPU1sEVniLMIyDAZWeHdV+NTA=
909	912		golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
910	913		golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
911	914		golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
		skipped 225 lines

■ ■ ■ ■ ■ ■

internal/string_helpers.go

		skipped 28 lines
29	29		return false
30	30		}
31	31
	32	+	func SplitAny(s string, seps string) []string {
	33	+	splitter := func(r rune) bool {
	34	+	return strings.ContainsRune(seps, r)
	35	+	}
	36	+	return strings.FieldsFunc(s, splitter)
	37	+	}
	38	+

■ ■ ■ ■ ■ ■

internal/string_helpers_test.go

		skipped 104 lines
105	105		}
106	106		}
107	107
	108	+	func TestSplitAny(t *testing.T) {
	109	+
	110	+	tests := []struct {
	111	+	name string
	112	+	input string
	113	+	fields string
	114	+	want []string
	115	+	}{
	116	+	{
	117	+	name: "simple",
	118	+	input: "a,b,c",
	119	+	fields: ",",
	120	+	want: []string{"a", "b", "c"},
	121	+	},
	122	+	{
	123	+	name: "empty",
	124	+	input: "",
	125	+	fields: ",",
	126	+	want: []string{},
	127	+	},
	128	+	{
	129	+	name: "multiple separators",
	130	+	input: "a,b\nc:d",
	131	+	fields: ",:\n",
	132	+	want: []string{"a", "b", "c", "d"},
	133	+	},
	134	+	}
	135	+	for _, tt := range tests {
	136	+	t.Run(tt.name, func(t *testing.T) {
	137	+	assert.Equal(t, tt.want, SplitAny(tt.input, tt.fields))
	138	+	})
	139	+	}
	140	+	}
	141	+

■ ■ ■ ■ ■ ■

syft/file/all_regular_files.go

1	1		package file
2	2
3	3		import (
	4	+	"github.com/anchore/stereoscope/pkg/file"
4	5		"github.com/anchore/syft/internal/log"
5	6		"github.com/anchore/syft/syft/source"
6	7		)
		skipped 13 lines
20	21		continue
21	22		}
22	23
23		-	if metadata.Type != source.RegularFile {
	24	+	if metadata.Type != file.TypeRegular {
24	25		continue
25	26		}
26	27		locations = append(locations, resolvedLocation)
		skipped 5 lines

■ ■ ■ ■ ■ ■

syft/file/all_regular_files_test.go

		skipped 2 lines
3	3		import (
4	4		"testing"
5	5
	6	+	"github.com/google/go-cmp/cmp"
6	7		"github.com/scylladb/go-set/strset"
7	8		"github.com/stretchr/testify/assert"
8	9		"github.com/stretchr/testify/require"
		skipped 60 lines
69	70		virtualLocations.Add(l.VirtualPath)
70	71		}
71	72		}
72		-	assert.ElementsMatch(t, tt.wantRealPaths.List(), realLocations.List(), "mismatched real paths")
73		-	assert.ElementsMatch(t, tt.wantVirtualPaths.List(), virtualLocations.List(), "mismatched virtual paths")
	73	+	assert.ElementsMatch(t, tt.wantRealPaths.List(), realLocations.List(), "real paths differ: "+cmp.Diff(tt.wantRealPaths.List(), realLocations.List()))
	74	+	assert.ElementsMatch(t, tt.wantVirtualPaths.List(), virtualLocations.List(), "virtual paths differ: "+cmp.Diff(tt.wantVirtualPaths.List(), virtualLocations.List()))
74	75		})
75	76		}
76	77		}
		skipped 1 lines

■ ■ ■ ■ ■ ■

syft/file/digest_cataloger.go

		skipped 10 lines
11	11		"github.com/wagoodman/go-partybus"
12	12		"github.com/wagoodman/go-progress"
13	13
	14	+	"github.com/anchore/stereoscope/pkg/file"
14	15		"github.com/anchore/syft/internal"
15	16		"github.com/anchore/syft/internal/bus"
16	17		"github.com/anchore/syft/internal/log"
		skipped 48 lines
65	66		}
66	67
67	68		// we should only attempt to report digests for files that are regular files (don't attempt to resolve links)
68		-	if meta.Type != source.RegularFile {
	69	+	if meta.Type != file.TypeRegular {
69	70		return nil, errUndigestableFile
70	71		}
71	72
		skipped 71 lines

■ ■ ■ ■ ■ ■

syft/file/digest_cataloger_test.go

		skipped 145 lines
146	146		if err != nil {
147	147		t.Fatalf("unable to get file=%q : %+v", test.path, err)
148	148		}
149		-	l := source.NewLocationFromImage(test.path, *ref, img)
	149	+	l := source.NewLocationFromImage(test.path, *ref.Reference, img)
150	150
151	151		if len(actual[l.Coordinates]) == 0 {
152	152		if test.expected != "" {
		skipped 10 lines

■ ■ ■ ■ ■ ■

syft/file/metadata_cataloger_test.go

		skipped 5 lines
6	6		"testing"
7	7
8	8		"github.com/stretchr/testify/assert"
	9	+	"github.com/stretchr/testify/require"
9	10
10	11		"github.com/anchore/stereoscope/pkg/file"
11	12		"github.com/anchore/stereoscope/pkg/imagetest"
		skipped 38 lines
50	51		path: "/file-1.txt",
51	52		exists: true,
52	53		expected: source.FileMetadata{
	54	+	Path: "/file-1.txt",
53	55		Mode: 0644,
54		-	Type: "RegularFile",
	56	+	Type: file.TypeRegular,
55	57		UserID: 1,
56	58		GroupID: 2,
57	59		Size: 7,
		skipped 4 lines
62	64		path: "/hardlink-1",
63	65		exists: true,
64	66		expected: source.FileMetadata{
	67	+	Path: "/hardlink-1",
65	68		Mode: 0644,
66		-	Type: "HardLink",
	69	+	Type: file.TypeHardLink,
67	70		LinkDestination: "file-1.txt",
68	71		UserID: 1,
69	72		GroupID: 2,
		skipped 4 lines
74	77		path: "/symlink-1",
75	78		exists: true,
76	79		expected: source.FileMetadata{
	80	+	Path: "/symlink-1",
77	81		Mode: 0777 \| os.ModeSymlink,
78		-	Type: "SymbolicLink",
	82	+	Type: file.TypeSymLink,
79	83		LinkDestination: "file-1.txt",
80	84		UserID: 0,
81	85		GroupID: 0,
		skipped 4 lines
86	90		path: "/char-device-1",
87	91		exists: true,
88	92		expected: source.FileMetadata{
	93	+	Path: "/char-device-1",
89	94		Mode: 0644 \| os.ModeDevice \| os.ModeCharDevice,
90		-	Type: "CharacterDevice",
	95	+	Type: file.TypeCharacterDevice,
91	96		UserID: 0,
92	97		GroupID: 0,
93	98		MIMEType: "",
		skipped 3 lines
97	102		path: "/block-device-1",
98	103		exists: true,
99	104		expected: source.FileMetadata{
	105	+	Path: "/block-device-1",
100	106		Mode: 0644 \| os.ModeDevice,
101		-	Type: "BlockDevice",
	107	+	Type: file.TypeBlockDevice,
102	108		UserID: 0,
103	109		GroupID: 0,
104	110		MIMEType: "",
		skipped 3 lines
108	114		path: "/fifo-1",
109	115		exists: true,
110	116		expected: source.FileMetadata{
	117	+	Path: "/fifo-1",
111	118		Mode: 0644 \| os.ModeNamedPipe,
112		-	Type: "FIFONode",
	119	+	Type: file.TypeFIFO,
113	120		UserID: 0,
114	121		GroupID: 0,
115	122		MIMEType: "",
		skipped 3 lines
119	126		path: "/bin",
120	127		exists: true,
121	128		expected: source.FileMetadata{
	129	+	Path: "/bin",
122	130		Mode: 0755 \| os.ModeDir,
123		-	Type: "Directory",
	131	+	Type: file.TypeDirectory,
124	132		UserID: 0,
125	133		GroupID: 0,
126	134		MIMEType: "",
	135	+	IsDir: true,
127	136		},
128	137		},
129	138		}
		skipped 1 lines
131	140		for _, test := range tests {
132	141		t.Run(test.path, func(t *testing.T) {
133	142		_, ref, err := img.SquashedTree().File(file.Path(test.path))
134		-	if err != nil {
135		-	t.Fatalf("unable to get file: %+v", err)
136		-	}
	143	+	require.NoError(t, err)
137	144
138		-	l := source.NewLocationFromImage(test.path, *ref, img)
	145	+	l := source.NewLocationFromImage(test.path, *ref.Reference, img)
139	146
140	147		assert.Equal(t, test.expected, actual[l.Coordinates], "mismatched metadata")
141	148
		skipped 5 lines

■ ■ ■ ■ ■ ■

syft/formats/cyclonedxjson/encoder_test.go

		skipped 4 lines
5	5		"regexp"
6	6		"testing"
7	7
8		-	"github.com/anchore/syft/syft/formats/common/testutils"
	8	+	"github.com/anchore/syft/syft/formats/internal/testutils"
9	9		)
10	10
11	11		var updateCycloneDx = flag.Bool("update-cyclonedx", false, "update the *.golden files for cyclone-dx encoders")
		skipped 36 lines

■ ■ ■ ■ ■ ■

syft/formats/cyclonedxxml/encoder_test.go

		skipped 4 lines
5	5		"regexp"
6	6		"testing"
7	7
8		-	"github.com/anchore/syft/syft/formats/common/testutils"
	8	+	"github.com/anchore/syft/syft/formats/internal/testutils"
9	9		)
10	10
11	11		var updateCycloneDx = flag.Bool("update-cyclonedx", false, "update the *.golden files for cyclone-dx encoders")
		skipped 42 lines

■ ■ ■ ■ ■ ■

syft/formats/formats.go

		skipped 43 lines
44	44		for _, f := range Formats() {
45	45		if err := f.Validate(bytes.NewReader(by)); err != nil {
46	46		if !errors.Is(err, sbom.ErrValidationNotSupported) {
47		-	log.Debugf("format %s returned err: %+v", f.ID(), err)
	47	+	log.WithFields("error", err).Tracef("format validation for %s failed", f.ID())
48	48		}
49	49		continue
50	50		}
		skipped 94 lines

■ ■ ■ ■ ■ ■

syft/formats/common/testutils/utils.go syft/formats/internal/testutils/utils.go

		skipped 167 lines
168	168		Name: "package-1",
169	169		Version: "1.0.1",
170	170		Locations: source.NewLocationSet(
171		-	source.NewLocationFromImage(string(ref1.RealPath), *ref1, img),
	171	+	source.NewLocationFromImage(string(ref1.RealPath), *ref1.Reference, img),
172	172		),
173	173		Type: pkg.PythonPkg,
174	174		FoundBy: "the-cataloger-1",
		skipped 13 lines
188	188		Name: "package-2",
189	189		Version: "2.0.1",
190	190		Locations: source.NewLocationSet(
191		-	source.NewLocationFromImage(string(ref2.RealPath), *ref2, img),
	191	+	source.NewLocationFromImage(string(ref2.RealPath), *ref2.Reference, img),
192	192		),
193	193		Type: pkg.DebPkg,
194	194		FoundBy: "the-cataloger-2",
		skipped 116 lines

■ ■ ■ ■ ■ ■

syft/formats/spdxjson/encoder_test.go

		skipped 4 lines
5	5		"regexp"
6	6		"testing"
7	7
8		-	"github.com/anchore/syft/syft/formats/common/testutils"
	8	+	"github.com/anchore/syft/syft/formats/internal/testutils"
9	9		)
10	10
11	11		var updateSpdxJson = flag.Bool("update-spdx-json", false, "update the *.golden files for spdx-json encoders")
		skipped 48 lines

■ ■ ■ ■ ■ ■

syft/formats/spdxtagvalue/encoder_test.go

		skipped 4 lines
5	5		"regexp"
6	6		"testing"
7	7
8		-	"github.com/anchore/syft/syft/formats/common/testutils"
	8	+	"github.com/anchore/syft/syft/formats/internal/testutils"
9	9		"github.com/anchore/syft/syft/pkg"
10	10		"github.com/anchore/syft/syft/sbom"
11	11		"github.com/anchore/syft/syft/source"
		skipped 86 lines

■ ■ ■ ■ ■ ■

syft/formats/syftjson/decoder_test.go

		skipped 7 lines
8	8		"github.com/go-test/deep"
9	9		"github.com/stretchr/testify/assert"
10	10
11		-	"github.com/anchore/syft/syft/formats/common/testutils"
	11	+	"github.com/anchore/syft/syft/formats/internal/testutils"
12	12		)
13	13
14	14		func TestEncodeDecodeCycle(t *testing.T) {
		skipped 38 lines

■ ■ ■ ■ ■ ■

syft/formats/syftjson/encoder_test.go

		skipped 4 lines
5	5		"regexp"
6	6		"testing"
7	7
	8	+	stereoFile "github.com/anchore/stereoscope/pkg/file"
8	9		"github.com/anchore/syft/syft/artifact"
9	10		"github.com/anchore/syft/syft/cpe"
10	11		"github.com/anchore/syft/syft/file"
11		-	"github.com/anchore/syft/syft/formats/common/testutils"
	12	+	"github.com/anchore/syft/syft/formats/internal/testutils"
12	13		"github.com/anchore/syft/syft/linux"
13	14		"github.com/anchore/syft/syft/pkg"
14	15		"github.com/anchore/syft/syft/sbom"
		skipped 92 lines
107	108		FileMetadata: map[source.Coordinates]source.FileMetadata{
108	109		source.NewLocation("/a/place").Coordinates: {
109	110		Mode: 0775,
110		-	Type: "directory",
	111	+	Type: stereoFile.TypeDirectory,
111	112		UserID: 0,
112	113		GroupID: 0,
113	114		},
114	115		source.NewLocation("/a/place/a").Coordinates: {
115	116		Mode: 0775,
116		-	Type: "regularFile",
	117	+	Type: stereoFile.TypeRegular,
117	118		UserID: 0,
118	119		GroupID: 0,
119	120		},
120	121		source.NewLocation("/b").Coordinates: {
121	122		Mode: 0775,
122		-	Type: "symbolicLink",
	123	+	Type: stereoFile.TypeSymLink,
123	124		LinkDestination: "/c",
124	125		UserID: 0,
125	126		GroupID: 0,
126	127		},
127	128		source.NewLocation("/b/place/b").Coordinates: {
128	129		Mode: 0644,
129		-	Type: "regularFile",
	130	+	Type: stereoFile.TypeRegular,
130	131		UserID: 1,
131	132		GroupID: 2,
132	133		},
		skipped 85 lines

■ ■ ■ ■ ■ ■

syft/formats/syftjson/model/file.go

		skipped 13 lines
14	14		}
15	15
16	16		type FileMetadataEntry struct {
17		-	Mode int `json:"mode"`
18		-	Type source.FileType `json:"type"`
19		-	LinkDestination string `json:"linkDestination,omitempty"`
20		-	UserID int `json:"userID"`
21		-	GroupID int `json:"groupID"`
22		-	MIMEType string `json:"mimeType"`
	17	+	Mode int `json:"mode"`
	18	+	Type string `json:"type"`
	19	+	LinkDestination string `json:"linkDestination,omitempty"`
	20	+	UserID int `json:"userID"`
	21	+	GroupID int `json:"groupID"`
	22	+	MIMEType string `json:"mimeType"`
23	23		}
24	24

■ ■ ■ ■ ■ ■

syft/formats/syftjson/test-fixtures/snapshot/TestDirectoryEncoder.golden

		skipped 88 lines
89	89		}
90	90		},
91	91		"schema": {
92		-	"version": "6.1.0",
93		-	"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-6.1.0.json"
	92	+	"version": "6.2.0",
	93	+	"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-6.2.0.json"
94	94		}
95	95		}
96	96

■ ■ ■ ■ ■ ■

syft/formats/syftjson/test-fixtures/snapshot/TestEncodeFullJSONDocument.golden

		skipped 77 lines
78	78		},
79	79		"metadata": {
80	80		"mode": 775,
81		-	"type": "directory",
	81	+	"type": "Directory",
82	82		"userID": 0,
83	83		"groupID": 0,
84	84		"mimeType": ""
		skipped 6 lines
91	91		},
92	92		"metadata": {
93	93		"mode": 775,
94		-	"type": "regularFile",
	94	+	"type": "RegularFile",
95	95		"userID": 0,
96	96		"groupID": 0,
97	97		"mimeType": ""
		skipped 13 lines
111	111		},
112	112		"metadata": {
113	113		"mode": 775,
114		-	"type": "symbolicLink",
	114	+	"type": "SymbolicLink",
115	115		"linkDestination": "/c",
116	116		"userID": 0,
117	117		"groupID": 0,
		skipped 7 lines
125	125		},
126	126		"metadata": {
127	127		"mode": 644,
128		-	"type": "regularFile",
	128	+	"type": "RegularFile",
129	129		"userID": 1,
130	130		"groupID": 2,
131	131		"mimeType": ""
		skipped 61 lines

■ ■ ■ ■ ■ ■

syft/formats/syftjson/test-fixtures/snapshot/TestImageEncoder.golden

		skipped 8 lines
9	9		"locations": [
10	10		{
11	11		"path": "/somefile-1.txt",
12		-	"layerID": "sha256:6afd1cb55939d87ba4c298907d0a53059bb3742c2d65314643e2464071cf0a2d"
	12	+	"layerID": "sha256:fb6beecb75b39f4bb813dbf177e501edd5ddb3e69bb45cedeb78c676ee1b7a59"
13	13		}
14	14		],
15	15		"licenses": [
		skipped 24 lines
40	40		"locations": [
41	41		{
42	42		"path": "/somefile-2.txt",
43		-	"layerID": "sha256:657997cff9a836139186239bdfe77250239a700d0ed97d57e101c295e8244319"
	43	+	"layerID": "sha256:319b588ce64253a87b533c8ed01cf0025e0eac98e7b516e12532957e1244fdec"
44	44		}
45	45		],
46	46		"licenses": [],
		skipped 17 lines
64	64		],
65	65		"artifactRelationships": [],
66	66		"source": {
67		-	"id": "c85f7ae1b0ac38342c1cf1a6f7ea2b4b1ddc49cd1b24219ebd05dc10b3303491",
	67	+	"id": "1a678f111c8ddc66fd82687bb024e0dd6af61314404937a80e810c0cf317b796",
68	68		"type": "image",
69	69		"target": {
70	70		"userInput": "user-image-input",
71		-	"imageID": "sha256:b5c0bfa8bcf70c75d92ebebbf76af667906d56e6fad50c37e7f93df824a64b79",
	71	+	"imageID": "sha256:3c51b06feb0cda8ee62d0e3755ef2a8496a6b71f8a55b245f07f31c4bb813d31",
72	72		"manifestDigest": "sha256:2731251dc34951c0e50fcc643b4c5f74922dad1a5d98f302b504cf46cd5d9368",
73	73		"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
74	74		"tags": [
		skipped 3 lines
78	78		"layers": [
79	79		{
80	80		"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
81		-	"digest": "sha256:6afd1cb55939d87ba4c298907d0a53059bb3742c2d65314643e2464071cf0a2d",
	81	+	"digest": "sha256:fb6beecb75b39f4bb813dbf177e501edd5ddb3e69bb45cedeb78c676ee1b7a59",
82	82		"size": 22
83	83		},
84	84		{
85	85		"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
86		-	"digest": "sha256:657997cff9a836139186239bdfe77250239a700d0ed97d57e101c295e8244319",
	86	+	"digest": "sha256:319b588ce64253a87b533c8ed01cf0025e0eac98e7b516e12532957e1244fdec",
87	87		"size": 16
88	88		}
89	89		],
90		-	"manifest": "eyJzY2hlbWFWZXJzaW9uIjoyLCJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi92bmQuZG9ja2VyLmRpc3RyaWJ1dGlvbi5tYW5pZmVzdC52Mitqc29uIiwiY29uZmlnIjp7Im1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL3ZuZC5kb2NrZXIuY29udGFpbmVyLmltYWdlLnYxK2pzb24iLCJzaXplIjo2NzMsImRpZ2VzdCI6InNoYTI1NjpiNWMwYmZhOGJjZjcwYzc1ZDkyZWJlYmJmNzZhZjY2NzkwNmQ1NmU2ZmFkNTBjMzdlN2Y5M2RmODI0YTY0Yjc5In0sImxheWVycyI6W3sibWVkaWFUeXBlIjoiYXBwbGljYXRpb24vdm5kLmRvY2tlci5pbWFnZS5yb290ZnMuZGlmZi50YXIuZ3ppcCIsInNpemUiOjIwNDgsImRpZ2VzdCI6InNoYTI1Njo2YWZkMWNiNTU5MzlkODdiYTRjMjk4OTA3ZDBhNTMwNTliYjM3NDJjMmQ2NTMxNDY0M2UyNDY0MDcxY2YwYTJkIn0seyJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi92bmQuZG9ja2VyLmltYWdlLnJvb3Rmcy5kaWZmLnRhci5nemlwIiwic2l6ZSI6MjA0OCwiZGlnZXN0Ijoic2hhMjU2OjY1Nzk5N2NmZjlhODM2MTM5MTg2MjM5YmRmZTc3MjUwMjM5YTcwMGQwZWQ5N2Q1N2UxMDFjMjk1ZTgyNDQzMTkifV19",
91		-	"config": "eyJhcmNoaXRlY3R1cmUiOiJhbWQ2NCIsImNvbmZpZyI6eyJFbnYiOlsiUEFUSD0vdXNyL2xvY2FsL3NiaW46L3Vzci9sb2NhbC9iaW46L3Vzci9zYmluOi91c3IvYmluOi9zYmluOi9iaW4iXSwiV29ya2luZ0RpciI6Ii8iLCJPbkJ1aWxkIjpudWxsfSwiY3JlYXRlZCI6IjIwMjItMDgtMjVUMTY6MjI6MDguODkxMzY0Mjc4WiIsImhpc3RvcnkiOlt7ImNyZWF0ZWQiOiIyMDIyLTA4LTI1VDE2OjIyOjA4Ljc2MzMzMDMyM1oiLCJjcmVhdGVkX2J5IjoiQUREIGZpbGUtMS50eHQgL3NvbWVmaWxlLTEudHh0ICMgYnVpbGRraXQiLCJjb21tZW50IjoiYnVpbGRraXQuZG9ja2VyZmlsZS52MCJ9LHsiY3JlYXRlZCI6IjIwMjItMDgtMjVUMTY6MjI6MDguODkxMzY0Mjc4WiIsImNyZWF0ZWRfYnkiOiJBREQgZmlsZS0yLnR4dCAvc29tZWZpbGUtMi50eHQgIyBidWlsZGtpdCIsImNvbW1lbnQiOiJidWlsZGtpdC5kb2NrZXJmaWxlLnYwIn1dLCJvcyI6ImxpbnV4Iiwicm9vdGZzIjp7InR5cGUiOiJsYXllcnMiLCJkaWZmX2lkcyI6WyJzaGEyNTY6NmFmZDFjYjU1OTM5ZDg3YmE0YzI5ODkwN2QwYTUzMDU5YmIzNzQyYzJkNjUzMTQ2NDNlMjQ2NDA3MWNmMGEyZCIsInNoYTI1Njo2NTc5OTdjZmY5YTgzNjEzOTE4NjIzOWJkZmU3NzI1MDIzOWE3MDBkMGVkOTdkNTdlMTAxYzI5NWU4MjQ0MzE5Il19fQ==",
	90	+	"manifest": "eyJzY2hlbWFWZXJzaW9uIjoyLCJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi92bmQuZG9ja2VyLmRpc3RyaWJ1dGlvbi5tYW5pZmVzdC52Mitqc29uIiwiY29uZmlnIjp7Im1lZGlhVHlwZSI6ImFwcGxpY2F0aW9uL3ZuZC5kb2NrZXIuY29udGFpbmVyLmltYWdlLnYxK2pzb24iLCJzaXplIjo2NzMsImRpZ2VzdCI6InNoYTI1NjozYzUxYjA2ZmViMGNkYThlZTYyZDBlMzc1NWVmMmE4NDk2YTZiNzFmOGE1NWIyNDVmMDdmMzFjNGJiODEzZDMxIn0sImxheWVycyI6W3sibWVkaWFUeXBlIjoiYXBwbGljYXRpb24vdm5kLmRvY2tlci5pbWFnZS5yb290ZnMuZGlmZi50YXIuZ3ppcCIsInNpemUiOjIwNDgsImRpZ2VzdCI6InNoYTI1NjpmYjZiZWVjYjc1YjM5ZjRiYjgxM2RiZjE3N2U1MDFlZGQ1ZGRiM2U2OWJiNDVjZWRlYjc4YzY3NmVlMWI3YTU5In0seyJtZWRpYVR5cGUiOiJhcHBsaWNhdGlvbi92bmQuZG9ja2VyLmltYWdlLnJvb3Rmcy5kaWZmLnRhci5nemlwIiwic2l6ZSI6MjA0OCwiZGlnZXN0Ijoic2hhMjU2OjMxOWI1ODhjZTY0MjUzYTg3YjUzM2M4ZWQwMWNmMDAyNWUwZWFjOThlN2I1MTZlMTI1MzI5NTdlMTI0NGZkZWMifV19",
	91	+	"config": "eyJhcmNoaXRlY3R1cmUiOiJhbWQ2NCIsImNvbmZpZyI6eyJFbnYiOlsiUEFUSD0vdXNyL2xvY2FsL3NiaW46L3Vzci9sb2NhbC9iaW46L3Vzci9zYmluOi91c3IvYmluOi9zYmluOi9iaW4iXSwiV29ya2luZ0RpciI6Ii8iLCJPbkJ1aWxkIjpudWxsfSwiY3JlYXRlZCI6IjIwMjItMDgtMDFUMjA6MDk6MjIuNTA5NDIxNzEyWiIsImhpc3RvcnkiOlt7ImNyZWF0ZWQiOiIyMDIyLTA4LTAxVDIwOjA5OjIyLjQ4Nzg5NTUxOVoiLCJjcmVhdGVkX2J5IjoiQUREIGZpbGUtMS50eHQgL3NvbWVmaWxlLTEudHh0ICMgYnVpbGRraXQiLCJjb21tZW50IjoiYnVpbGRraXQuZG9ja2VyZmlsZS52MCJ9LHsiY3JlYXRlZCI6IjIwMjItMDgtMDFUMjA6MDk6MjIuNTA5NDIxNzEyWiIsImNyZWF0ZWRfYnkiOiJBREQgZmlsZS0yLnR4dCAvc29tZWZpbGUtMi50eHQgIyBidWlsZGtpdCIsImNvbW1lbnQiOiJidWlsZGtpdC5kb2NrZXJmaWxlLnYwIn1dLCJvcyI6ImxpbnV4Iiwicm9vdGZzIjp7InR5cGUiOiJsYXllcnMiLCJkaWZmX2lkcyI6WyJzaGEyNTY6ZmI2YmVlY2I3NWIzOWY0YmI4MTNkYmYxNzdlNTAxZWRkNWRkYjNlNjliYjQ1Y2VkZWI3OGM2NzZlZTFiN2E1OSIsInNoYTI1NjozMTliNTg4Y2U2NDI1M2E4N2I1MzNjOGVkMDFjZjAwMjVlMGVhYzk4ZTdiNTE2ZTEyNTMyOTU3ZTEyNDRmZGVjIl19fQ==",
92	92		"repoDigests": [],
93	93		"architecture": "",
94	94		"os": ""
		skipped 17 lines
112	112		}
113	113		},
114	114		"schema": {
115		-	"version": "6.1.0",
116		-	"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-6.1.0.json"
	115	+	"version": "6.2.0",
	116	+	"url": "https://raw.githubusercontent.com/anchore/syft/main/schema/json/schema-6.2.0.json"
117	117		}
118	118		}
119	119

syft/formats/syftjson/test-fixtures/snapshot/stereoscope-fixture-image-simple.golden

Binary file.

■ ■ ■ ■ ■ ■ ■

syft/formats/syftjson/to_format_model.go

		skipped 4 lines
5	5		"sort"
6	6		"strconv"
7	7
	8	+	stereoscopeFile "github.com/anchore/stereoscope/pkg/file"
8	9		"github.com/anchore/syft/internal"
9	10		"github.com/anchore/syft/internal/log"
10	11		"github.com/anchore/syft/syft/artifact"
		skipped 126 lines
137	138
138	139		return &model.FileMetadataEntry{
139	140		Mode: mode,
140		-	Type: metadata.Type,
	141	+	Type: toFileType(metadata.Type),
141	142		LinkDestination: metadata.LinkDestination,
142	143		UserID: metadata.UserID,
143	144		GroupID: metadata.GroupID,
144	145		MIMEType: metadata.MIMEType,
	146	+	}
	147	+	}
	148	+
	149	+	func toFileType(ty stereoscopeFile.Type) string {
	150	+	switch ty {
	151	+	case stereoscopeFile.TypeSymLink:
	152	+	return "SymbolicLink"
	153	+	case stereoscopeFile.TypeHardLink:
	154	+	return "HardLink"
	155	+	case stereoscopeFile.TypeDirectory:
	156	+	return "Directory"
	157	+	case stereoscopeFile.TypeSocket:
	158	+	return "Socket"
	159	+	case stereoscopeFile.TypeBlockDevice:
	160	+	return "BlockDevice"
	161	+	case stereoscopeFile.TypeCharacterDevice:
	162	+	return "CharacterDevice"
	163	+	case stereoscopeFile.TypeFIFO:
	164	+	return "FIFONode"
	165	+	case stereoscopeFile.TypeRegular:
	166	+	return "RegularFile"
	167	+	case stereoscopeFile.TypeIrregular:
	168	+	return "IrregularFile"
	169	+	default:
	170	+	return "Unknown"
145	171		}
146	172		}
147	173
		skipped 105 lines

■ ■ ■ ■ ■ ■

syft/formats/syftjson/to_format_model_test.go

		skipped 7 lines
8	8		"github.com/stretchr/testify/assert"
9	9		"github.com/stretchr/testify/require"
10	10
	11	+	"github.com/anchore/stereoscope/pkg/file"
11	12		"github.com/anchore/syft/internal"
12	13		"github.com/anchore/syft/syft/formats/syftjson/model"
13	14		"github.com/anchore/syft/syft/source"
		skipped 84 lines
98	99		assert.ElementsMatch(t, allSchemes.List(), testedSchemes.List(), "not all source.Schemes are under test")
99	100		}
100	101
	102	+	func Test_toFileType(t *testing.T) {
	103	+
	104	+	badType := file.Type(0x1337)
	105	+	var allTypesTested []file.Type
	106	+	tests := []struct {
	107	+	ty file.Type
	108	+	name string
	109	+	}{
	110	+	{
	111	+	ty: file.TypeRegular,
	112	+	name: "RegularFile",
	113	+	},
	114	+	{
	115	+	ty: file.TypeDirectory,
	116	+	name: "Directory",
	117	+	},
	118	+	{
	119	+	ty: file.TypeSymLink,
	120	+	name: "SymbolicLink",
	121	+	},
	122	+	{
	123	+	ty: file.TypeHardLink,
	124	+	name: "HardLink",
	125	+	},
	126	+	{
	127	+	ty: file.TypeSocket,
	128	+	name: "Socket",
	129	+	},
	130	+	{
	131	+	ty: file.TypeCharacterDevice,
	132	+	name: "CharacterDevice",
	133	+	},
	134	+	{
	135	+	ty: file.TypeBlockDevice,
	136	+	name: "BlockDevice",
	137	+	},
	138	+	{
	139	+	ty: file.TypeFIFO,
	140	+	name: "FIFONode",
	141	+	},
	142	+	{
	143	+	ty: file.TypeIrregular,
	144	+	name: "IrregularFile",
	145	+	},
	146	+	{
	147	+	ty: badType,
	148	+	name: "Unknown",
	149	+	},
	150	+	}
	151	+	for _, tt := range tests {
	152	+	t.Run(tt.name, func(t *testing.T) {
	153	+	assert.Equalf(t, tt.name, toFileType(tt.ty), "toFileType(%v)", tt.ty)
	154	+	if tt.ty != badType {
	155	+	allTypesTested = append(allTypesTested, tt.ty)
	156	+	}
	157	+	})
	158	+	}
	159	+
	160	+	assert.ElementsMatch(t, allTypesTested, file.AllTypes(), "not all file.Types are under test")
	161	+	}
	162	+

■ ■ ■ ■ ■ ■

syft/formats/table/encoder_test.go

		skipped 5 lines
6	6
7	7		"github.com/go-test/deep"
8	8
9		-	"github.com/anchore/syft/syft/formats/common/testutils"
	9	+	"github.com/anchore/syft/syft/formats/internal/testutils"
10	10		)
11	11
12	12		var updateTableGoldenFiles = flag.Bool("update-table", false, "update the *.golden files for table format")
		skipped 39 lines

■ ■ ■ ■ ■ ■

syft/formats/template/encoder_test.go

		skipped 5 lines
6	6
7	7		"github.com/stretchr/testify/assert"
8	8
9		-	"github.com/anchore/syft/syft/formats/common/testutils"
	9	+	"github.com/anchore/syft/syft/formats/internal/testutils"
10	10		)
11	11
12	12		var updateTmpl = flag.Bool("update-tmpl", false, "update the *.golden files for json encoders")
		skipped 20 lines

■ ■ ■ ■ ■ ■

syft/formats/text/encoder_test.go

		skipped 3 lines
4	4		"flag"
5	5		"testing"
6	6
7		-	"github.com/anchore/syft/syft/formats/common/testutils"
	7	+	"github.com/anchore/syft/syft/formats/internal/testutils"
8	8		)
9	9
10	10		var updateTextEncoderGoldenFiles = flag.Bool("update-text", false, "update the *.golden files for text encoder")
		skipped 21 lines

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/alpm/cataloger_test.go

1	+	package alpm
2	+
3	+	import (
4	+	"testing"
5	+
6	+	"github.com/google/go-cmp/cmp/cmpopts"
7	+
8	+	"github.com/anchore/syft/syft/artifact"
9	+	"github.com/anchore/syft/syft/file"
10	+	"github.com/anchore/syft/syft/pkg"
11	+	"github.com/anchore/syft/syft/pkg/cataloger/internal/pkgtest"
12	+	"github.com/anchore/syft/syft/source"
13	+	)
14	+
15	+	func TestAlpmCataloger(t *testing.T) {
16	+
17	+	expectedPkgs := []pkg.Package{
18	+	{
19	+	Name: "gmp",
20	+	Version: "6.2.1-2",
21	+	Type: pkg.AlpmPkg,
22	+	FoundBy: "alpmdb-cataloger",
23	+	Licenses: []string{"LGPL3", "GPL"},
24	+	Locations: source.NewLocationSet(source.NewLocation("var/lib/pacman/local/gmp-6.2.1-2/desc")),
25	+	CPEs: nil,
26	+	PURL: "",
27	+	MetadataType: "AlpmMetadata",
28	+	Metadata: pkg.AlpmMetadata{
29	+	BasePackage: "gmp",
30	+	Package: "gmp",
31	+	Version: "6.2.1-2",
32	+	Description: "A free library for arbitrary precision arithmetic",
33	+	Architecture: "x86_64",
34	+	Size: 1044438,
35	+	Packager: "Antonio Rojas <[email protected]>",
36	+	License: "LGPL3\nGPL",
37	+	URL: "https://gmplib.org/",
38	+	Validation: "pgp",
39	+	Reason: 1,
40	+	Files: []pkg.AlpmFileRecord{
41	+	{
42	+	Path: "/usr",
43	+	Type: "dir",
44	+	Digests: []file.Digest{},
45	+	},
46	+	{
47	+	Path: "/usr/include",
48	+	Type: "dir",
49	+	Digests: []file.Digest{},
50	+	},
51	+	{
52	+	Path: "/usr/include/gmp.h",
53	+	Size: "84140",
54	+	Digests: []file.Digest{
55	+	{Algorithm: "md5", Value: "76595f70565c72550eb520809bf86856"},
56	+	{Algorithm: "sha256", Value: "91a614b9202453153fe3b7512d15e89659108b93ce8841c8e13789eb85da9e3a"},
57	+	},
58	+	},
59	+	{
60	+	Path: "/usr/include/gmpxx.h",
61	+	Size: "129113",
62	+	Digests: []file.Digest{
63	+	{Algorithm: "md5", Value: "ea3d21de4bcf7c696799c5c55dd3655b"},
64	+	{Algorithm: "sha256", Value: "0011ae411a0bc1030e07d968b32fdc1343f5ac2a17b7d28f493e7976dde2ac82"},
65	+	},
66	+	},
67	+	{
68	+	Path: "/usr/lib",
69	+	Type: "dir",
70	+	Digests: []file.Digest{},
71	+	},
72	+	{
73	+	Path: "/usr/lib/libgmp.so",
74	+	Type: "link",
75	+	Link: "libgmp.so.10.4.1",
76	+	Digests: []file.Digest{},
77	+	},
78	+	{
79	+	Path: "/usr/lib/libgmp.so.10",
80	+	Type: "link",
81	+	Link: "libgmp.so.10.4.1",
82	+	Digests: []file.Digest{},
83	+	},
84	+	{
85	+	Path: "/usr/lib/libgmp.so.10.4.1",
86	+	Size: "663224",
87	+	Digests: []file.Digest{
88	+	{Algorithm: "md5", Value: "d6d03eadacdd9048d5b2adf577e9d722"},
89	+	{Algorithm: "sha256", Value: "39898bd3d8d6785222432fa8b8aef7ce3b7e5bbfc66a52b7c0da09bed4adbe6a"},
90	+	},
91	+	},
92	+	{
93	+	Path: "/usr/lib/libgmpxx.so",
94	+	Type: "link",
95	+	Link: "libgmpxx.so.4.6.1",
96	+	Digests: []file.Digest{},
97	+	},
98	+	{
99	+	Path: "/usr/lib/libgmpxx.so.4",
100	+	Type: "link",
101	+	Link: "libgmpxx.so.4.6.1",
102	+	Digests: []file.Digest{},
103	+	},
104	+	{
105	+	Path: "/usr/lib/libgmpxx.so.4.6.1",
106	+	Size: "30680",
107	+	Digests: []file.Digest{
108	+	{Algorithm: "md5", Value: "dd5f0c4d635fa599fa7f4339c0e8814d"},
109	+	{Algorithm: "sha256", Value: "0ef67cbde4841f58d2e4b41f59425eb87c9eeaf4e649c060b326342c53bedbec"},
110	+	},
111	+	},
112	+	{
113	+	Path: "/usr/lib/pkgconfig",
114	+	Type: "dir",
115	+	Digests: []file.Digest{},
116	+	},
117	+	{
118	+	Path: "/usr/lib/pkgconfig/gmp.pc",
119	+	Size: "245",
120	+	Digests: []file.Digest{
121	+	{Algorithm: "md5", Value: "a91a9f1b66218cb77b9cd2cdf341756d"},
122	+	{Algorithm: "sha256", Value: "4e9de547a48c4e443781e9fa702a1ec5a23ee28b4bc520306cff2541a855be37"},
123	+	},
124	+	},
125	+	{
126	+	Path: "/usr/lib/pkgconfig/gmpxx.pc",
127	+	Size: "280",
128	+	Digests: []file.Digest{
129	+	{Algorithm: "md5", Value: "8c0f54e987934352177a6a30a811b001"},
130	+	{Algorithm: "sha256", Value: "fc5dbfbe75977057ba50953d94b9daecf696c9fdfe5b94692b832b44ecca871b"},
131	+	},
132	+	},
133	+	{
134	+	Path: "/usr/share",
135	+	Type: "dir",
136	+	Digests: []file.Digest{},
137	+	},
138	+	{
139	+	Path: "/usr/share/info",
140	+	Type: "dir",
141	+	Digests: []file.Digest{},
142	+	},
143	+	{
144	+	Path: "/usr/share/info/gmp.info-1.gz",
145	+	Size: "85892",
146	+	Digests: []file.Digest{
147	+	{Algorithm: "md5", Value: "63304d4d2f0247fb8a999fae66a81c19"},
148	+	{Algorithm: "sha256", Value: "86288c1531a2789db5da8b9838b5cde4db07bda230ae11eba23a1f33698bd14e"},
149	+	},
150	+	},
151	+	{
152	+	Path: "/usr/share/info/gmp.info-2.gz",
153	+	Size: "48484",
154	+	Digests: []file.Digest{
155	+	{Algorithm: "md5", Value: "4bb0dadec416d305232cac6eae712ff7"},
156	+	{Algorithm: "sha256", Value: "b7443c1b529588d98a074266087f79b595657ac7274191c34b10a9ceedfa950e"},
157	+	},
158	+	},
159	+	{
160	+	Path: "/usr/share/info/gmp.info.gz",
161	+	Size: "2380",
162	+	Digests: []file.Digest{
163	+	{Algorithm: "md5", Value: "cf6880fb0d862ee1da0d13c3831b5720"},
164	+	{Algorithm: "sha256", Value: "a13c8eecda3f3e5ad1e09773e47a9686f07d9d494eaddf326f3696bbef1548fd"},
165	+	},
166	+	},
167	+	},
168	+	Backup: []pkg.AlpmFileRecord{},
169	+	},
170	+	},
171	+	}
172	+
173	+	// TODO: relationships are not under test yet
174	+	var expectedRelationships []artifact.Relationship
175	+
176	+	pkgtest.NewCatalogTester().
177	+	FromDirectory(t, "test-fixtures/gmp-fixture").
178	+	WithCompareOptions(cmpopts.IgnoreFields(pkg.AlpmFileRecord{}, "Time")).
179	+	Expects(expectedPkgs, expectedRelationships).
180	+	TestCataloger(t, NewAlpmdbCataloger())
181	+
182	+	}
183	+
184	+	func TestCataloger_Globs(t *testing.T) {
185	+	tests := []struct {
186	+	name string
187	+	fixture string
188	+	expected []string
189	+	}{
190	+	{
191	+	name: "obtain description files",
192	+	fixture: "test-fixtures/glob-paths",
193	+	expected: []string{
194	+	"var/lib/pacman/local/base-1.0/desc",
195	+	"var/lib/pacman/local/dive-0.10.0/desc",
196	+	},
197	+	},
198	+	}
199	+
200	+	for _, test := range tests {
201	+	t.Run(test.name, func(t *testing.T) {
202	+	pkgtest.NewCatalogTester().
203	+	FromDirectory(t, test.fixture).
204	+	ExpectsResolverContentQueries(test.expected).
205	+	IgnoreUnfulfilledPathResponses("var/lib/pacman/local/base-1.0/mtree", "var/lib/pacman/local/dive-0.10.0/mtree").
206	+	TestCataloger(t, NewAlpmdbCataloger())
207	+	})
208	+	}
209	+	}
210	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/alpm/package.go

1	1		package alpm
2	2
3	3		import (
4		-	"strings"
5		-
6	4		"github.com/anchore/packageurl-go"
	5	+	"github.com/anchore/syft/internal"
7	6		"github.com/anchore/syft/syft/linux"
8	7		"github.com/anchore/syft/syft/pkg"
9	8		"github.com/anchore/syft/syft/source"
		skipped 5 lines
15	14		Version: m.Version,
16	15		Locations: source.NewLocationSet(locations...),
17	16		Type: pkg.AlpmPkg,
18		-	Licenses: strings.Split(m.License, " "),
	17	+	Licenses: internal.SplitAny(m.License, " \n"),
19	18		PURL: packageURL(m, release),
20	19		MetadataType: pkg.AlpmMetadataType,
21	20		Metadata: m,
		skipped 32 lines

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/alpm/parse_alpm_db.go

		skipped 64 lines
65	65		metadata.Backup = filesMetadata.Backup
66	66		}
67	67
	68	+	if metadata.Package == "" {
	69	+	return nil, nil, nil
	70	+	}
	71	+
68	72		return []pkg.Package{
69	73		newPackage(*metadata, env.LinuxRelease, reader.Location),
70	74		}, nil, nil
		skipped 170 lines

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/alpm/parse_alpm_db_test.go

		skipped 15 lines
16	16		func TestDatabaseParser(t *testing.T) {
17	17		tests := []struct {
18	18		name string
	19	+	fixture string
19	20		expected pkg.AlpmMetadata
20	21		}{
21	22		{
22		-	name: "test alpm database parsing",
	23	+	name: "test alpm database parsing",
	24	+	fixture: "test-fixtures/files",
23	25		expected: pkg.AlpmMetadata{
24	26		Backup: []pkg.AlpmFileRecord{
25	27		{
		skipped 64 lines
90	92
91	93		for _, test := range tests {
92	94		t.Run(test.name, func(t *testing.T) {
93		-	f, err := os.Open("test-fixtures/files")
	95	+	f, err := os.Open(test.fixture)
94	96		require.NoError(t, err)
95	97		t.Cleanup(func() { require.NoError(t, f.Close()) })
96	98
		skipped 79 lines

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/alpm/test-fixtures/glob-paths/var/lib/pacman/local/base-1.0/desc

1

+

bogus desc file

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/alpm/test-fixtures/glob-paths/var/lib/pacman/local/base-1.0/files

1

+

bogus files

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/alpm/test-fixtures/glob-paths/var/lib/pacman/local/dive-0.10.0/desc

1

+

bogus desc file

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/alpm/test-fixtures/gmp-fixture/var/lib/pacman/local/gmp-6.2.1-2/desc

1	+	%NAME%
2	+	gmp
3	+
4	+	%VERSION%
5	+	6.2.1-2
6	+
7	+	%BASE%
8	+	gmp
9	+
10	+	%DESC%
11	+	A free library for arbitrary precision arithmetic
12	+
13	+	%URL%
14	+	https://gmplib.org/
15	+
16	+	%ARCH%
17	+	x86_64
18	+
19	+	%BUILDDATE%
20	+	1653121258
21	+
22	+	%INSTALLDATE%
23	+	1665878640
24	+
25	+	%PACKAGER%
26	+	Antonio Rojas <[email protected]>
27	+
28	+	%SIZE%
29	+	1044438
30	+
31	+	%REASON%
32	+	1
33	+
34	+	%LICENSE%
35	+	LGPL3
36	+	GPL
37	+
38	+	%VALIDATION%
39	+	pgp
40	+
41	+	%DEPENDS%
42	+	gcc-libs
43	+	sh
44	+
45	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/alpm/test-fixtures/gmp-fixture/var/lib/pacman/local/gmp-6.2.1-2/files

1	+	%FILES%
2	+	usr/
3	+	usr/include/
4	+	usr/include/gmp.h
5	+	usr/include/gmpxx.h
6	+	usr/lib/
7	+	usr/lib/libgmp.so
8	+	usr/lib/libgmp.so.10
9	+	usr/lib/libgmp.so.10.4.1
10	+	usr/lib/libgmpxx.so
11	+	usr/lib/libgmpxx.so.4
12	+	usr/lib/libgmpxx.so.4.6.1
13	+	usr/lib/pkgconfig/
14	+	usr/lib/pkgconfig/gmp.pc
15	+	usr/lib/pkgconfig/gmpxx.pc
16	+	usr/share/
17	+	usr/share/info/
18	+	usr/share/info/gmp.info-1.gz
19	+	usr/share/info/gmp.info-2.gz
20	+	usr/share/info/gmp.info.gz
21	+
22	+

syft/pkg/cataloger/alpm/test-fixtures/gmp-fixture/var/lib/pacman/local/gmp-6.2.1-2/mtree

Binary file.

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/apkdb/cataloger_test.go

1	+	package apkdb
2	+
3	+	import (
4	+	"testing"
5	+
6	+	"github.com/anchore/syft/syft/pkg/cataloger/internal/pkgtest"
7	+	)
8	+
9	+	func TestCataloger_Globs(t *testing.T) {
10	+	tests := []struct {
11	+	name string
12	+	fixture string
13	+	expected []string
14	+	}{
15	+	{
16	+	name: "obtain DB files",
17	+	fixture: "test-fixtures/glob-paths",
18	+	expected: []string{"lib/apk/db/installed"},
19	+	},
20	+	}
21	+
22	+	for _, test := range tests {
23	+	t.Run(test.name, func(t *testing.T) {
24	+	pkgtest.NewCatalogTester().
25	+	FromDirectory(t, test.fixture).
26	+	ExpectsResolverContentQueries(test.expected).
27	+	TestCataloger(t, NewApkdbCataloger())
28	+	})
29	+	}
30	+	}
31	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/apkdb/parse_apk_db.go

		skipped 6 lines
7	7		"strconv"
8	8		"strings"
9	9
	10	+	"github.com/anchore/syft/internal"
10	11		"github.com/anchore/syft/internal/log"
11	12		"github.com/anchore/syft/syft/artifact"
12	13		"github.com/anchore/syft/syft/file"
		skipped 337 lines
350	351		return relationships
351	352		}
352	353
353		-	func splitAny(s string, seps string) []string {
354		-	splitter := func(r rune) bool {
355		-	return strings.ContainsRune(seps, r)
356		-	}
357		-	return strings.FieldsFunc(s, splitter)
358		-	}
359		-
360	354		func stripVersionSpecifier(s string) string {
361	355		// examples:
362	356		// musl>=1 --> musl
363	357		// cmd:scanelf=1.3.4-r0 --> cmd:scanelf
364	358
365		-	items := splitAny(s, "<>=")
	359	+	items := internal.SplitAny(s, "<>=")
366	360		if len(items) == 0 {
367	361		return s
368	362		}
		skipped 4 lines

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/apkdb/test-fixtures/glob-paths/lib/apk/db/installed

1

+

bogus db contents

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/binary/cataloger.go

		skipped 31 lines
32	32		var relationships []artifact.Relationship
33	33
34	34		for _, cls := range defaultClassifiers {
	35	+	log.WithFields("classifier", cls.Class).Trace("cataloging binaries")
35	36		pkgs, err := catalog(resolver, cls)
36	37		if err != nil {
37	38		log.WithFields("error", err, "classifier", cls.Class).Warn("unable to catalog binary package: %w", err)
		skipped 54 lines

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/binary/cataloger_test.go

		skipped 12 lines
13	13		"github.com/anchore/syft/syft/source"
14	14		)
15	15
16		-	func TestClassifierCataloger_DefaultClassifiers_PositiveCases(t *testing.T) {
	16	+	func Test_Cataloger_DefaultClassifiers_PositiveCases(t *testing.T) {
17	17		tests := []struct {
18	18		name string
19	19		fixtureDir string
		skipped 420 lines
440	440		}
441	441		}
442	442
443		-	func TestClassifierCataloger_DefaultClassifiers_PositiveCases_Image(t *testing.T) {
	443	+	func Test_Cataloger_DefaultClassifiers_PositiveCases_Image(t *testing.T) {
444	444		tests := []struct {
445	445		name string
446	446		fixtureImage string
		skipped 74 lines
521	521		}
522	522
523	523		type panicyResolver struct {
524		-	globCalled bool
	524	+	searchCalled bool
525	525		}
526	526
527		-	func (p panicyResolver) FileContentsByLocation(location source.Location) (io.ReadCloser, error) {
	527	+	func (p *panicyResolver) FilesByExtension(_ ...string) ([]source.Location, error) {
	528	+	p.searchCalled = true
528	529		return nil, errors.New("not implemented")
529	530		}
530	531
531		-	func (p panicyResolver) HasPath(s string) bool {
	532	+	func (p *panicyResolver) FilesByBasename(_ ...string) ([]source.Location, error) {
	533	+	p.searchCalled = true
	534	+	return nil, errors.New("not implemented")
	535	+	}
	536	+
	537	+	func (p *panicyResolver) FilesByBasenameGlob(_ ...string) ([]source.Location, error) {
	538	+	p.searchCalled = true
	539	+	return nil, errors.New("not implemented")
	540	+	}
	541	+
	542	+	func (p *panicyResolver) FileContentsByLocation(_ source.Location) (io.ReadCloser, error) {
	543	+	p.searchCalled = true
	544	+	return nil, errors.New("not implemented")
	545	+	}
	546	+
	547	+	func (p *panicyResolver) HasPath(s string) bool {
532	548		return true
533	549		}
534	550
535		-	func (p panicyResolver) FilesByPath(paths ...string) ([]source.Location, error) {
	551	+	func (p *panicyResolver) FilesByPath(_ ...string) ([]source.Location, error) {
	552	+	p.searchCalled = true
536	553		return nil, errors.New("not implemented")
537	554		}
538	555
539		-	func (p *panicyResolver) FilesByGlob(patterns ...string) ([]source.Location, error) {
540		-	p.globCalled = true
	556	+	func (p *panicyResolver) FilesByGlob(_ ...string) ([]source.Location, error) {
	557	+	p.searchCalled = true
541	558		return nil, errors.New("not implemented")
542	559		}
543	560
544		-	func (p panicyResolver) FilesByMIMEType(types ...string) ([]source.Location, error) {
	561	+	func (p *panicyResolver) FilesByMIMEType(_ ...string) ([]source.Location, error) {
	562	+	p.searchCalled = true
545	563		return nil, errors.New("not implemented")
546	564		}
547	565
548		-	func (p panicyResolver) RelativeFileByPath(_ source.Location, path string) *source.Location {
	566	+	func (p panicyResolver) RelativeFileByPath(_ source.Location, _ string) source.Location {
549	567		return nil
550	568		}
551	569
552		-	func (p panicyResolver) AllLocations() <-chan source.Location {
	570	+	func (p *panicyResolver) AllLocations() <-chan source.Location {
553	571		return nil
554	572		}
555	573
556		-	func (p panicyResolver) FileMetadataByLocation(location source.Location) (source.FileMetadata, error) {
	574	+	func (p *panicyResolver) FileMetadataByLocation(_ source.Location) (source.FileMetadata, error) {
557	575		return source.FileMetadata{}, errors.New("not implemented")
558	576		}
559	577
		skipped 3 lines
563	581		resolver := &panicyResolver{}
564	582		_, _, err := c.Catalog(resolver)
565	583		assert.NoError(t, err)
566		-	assert.True(t, resolver.globCalled)
	584	+	assert.True(t, resolver.searchCalled)
567	585		}
568	586

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/binary/classifier_test.go

		skipped 20 lines
21	21		fixture: "test-fixtures/version.txt",
22	22		classifier: classifier{
23	23		Package: "some-app",
24		-	FileGlob: ".*/version.txt",
	24	+	FileGlob: "**/version.txt",
25	25		EvidenceMatcher: fileContentsVersionMatcher(`(?m)my-verison:(?P<version>[0-9.]+)`),
26	26		CPEs: []cpe.CPE{},
27	27		},
		skipped 4 lines
32	32		fixture: "test-fixtures/version.txt",
33	33		classifier: classifier{
34	34		Package: "some-app",
35		-	FileGlob: ".*/version.txt",
	35	+	FileGlob: "**/version.txt",
36	36		EvidenceMatcher: fileContentsVersionMatcher(`(?m)my-verison:(?P<version>[0-9.]+)`),
37	37		CPEs: []cpe.CPE{
38	38		cpe.Must("cpe:2.3:a:some:app::::::::"),
		skipped 8 lines
47	47		fixture: "test-fixtures/version.txt",
48	48		classifier: classifier{
49	49		Package: "some-app",
50		-	FileGlob: ".*/version.txt",
	50	+	FileGlob: "**/version.txt",
51	51		EvidenceMatcher: fileContentsVersionMatcher(`(?m)my-verison:(?P<version>[0-9.]+)`),
52	52		CPEs: []cpe.CPE{
53	53		cpe.Must("cpe:2.3:a:some:app::::::::"),
		skipped 35 lines

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/cataloger.go

		skipped 40 lines
41	41		alpm.NewAlpmdbCataloger(),
42	42		ruby.NewGemSpecCataloger(),
43	43		python.NewPythonPackageCataloger(),
44		-	php.NewPHPComposerInstalledCataloger(),
45		-	javascript.NewJavascriptPackageCataloger(),
	44	+	php.NewComposerInstalledCataloger(),
	45	+	javascript.NewPackageCataloger(),
46	46		deb.NewDpkgdbCataloger(),
47	47		rpm.NewRpmDBCataloger(),
48	48		java.NewJavaCataloger(cfg.Java()),
		skipped 14 lines
63	63		ruby.NewGemFileLockCataloger(),
64	64		python.NewPythonIndexCataloger(),
65	65		python.NewPythonPackageCataloger(),
66		-	php.NewPHPComposerLockCataloger(),
67		-	javascript.NewJavascriptLockCataloger(),
	66	+	php.NewComposerLockCataloger(),
	67	+	javascript.NewLockCataloger(),
68	68		deb.NewDpkgdbCataloger(),
69	69		rpm.NewRpmDBCataloger(),
70	70		rpm.NewFileCataloger(),
		skipped 25 lines
96	96		ruby.NewGemSpecCataloger(),
97	97		python.NewPythonIndexCataloger(),
98	98		python.NewPythonPackageCataloger(),
99		-	javascript.NewJavascriptLockCataloger(),
100		-	javascript.NewJavascriptPackageCataloger(),
	99	+	javascript.NewLockCataloger(),
	100	+	javascript.NewPackageCataloger(),
101	101		deb.NewDpkgdbCataloger(),
102	102		rpm.NewRpmDBCataloger(),
103	103		rpm.NewFileCataloger(),
		skipped 7 lines
111	111		rust.NewAuditBinaryCataloger(),
112	112		dart.NewPubspecLockCataloger(),
113	113		dotnet.NewDotnetDepsCataloger(),
114		-	php.NewPHPComposerInstalledCataloger(),
115		-	php.NewPHPComposerLockCataloger(),
	114	+	php.NewComposerInstalledCataloger(),
	115	+	php.NewComposerLockCataloger(),
116	116		swift.NewCocoapodsCataloger(),
117	117		cpp.NewConanCataloger(),
118	118		portage.NewPortageCataloger(),
		skipped 52 lines

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/cpp/cataloger_test.go

1	+	package cpp
2	+
3	+	import (
4	+	"testing"
5	+
6	+	"github.com/anchore/syft/syft/pkg/cataloger/internal/pkgtest"
7	+	)
8	+
9	+	func TestCataloger_Globs(t *testing.T) {
10	+	tests := []struct {
11	+	name string
12	+	fixture string
13	+	expected []string
14	+	}{
15	+	{
16	+	name: "obtain conan files",
17	+	fixture: "test-fixtures/glob-paths",
18	+	expected: []string{
19	+	"somewhere/src/conanfile.txt",
20	+	"somewhere/src/conan.lock",
21	+	},
22	+	},
23	+	}
24	+
25	+	for _, test := range tests {
26	+	t.Run(test.name, func(t *testing.T) {
27	+	pkgtest.NewCatalogTester().
28	+	FromDirectory(t, test.fixture).
29	+	ExpectsResolverContentQueries(test.expected).
30	+	TestCataloger(t, NewConanCataloger())
31	+	})
32	+	}
33	+	}
34	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/cpp/test-fixtures/glob-paths/somewhere/src/conan.lock

1

+

bogus conan.lock

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/cpp/test-fixtures/glob-paths/somewhere/src/conanfile.txt

1

+

bogus conan file

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/dart/cataloger_test.go

1	+	package dart
2	+
3	+	import (
4	+	"testing"
5	+
6	+	"github.com/anchore/syft/syft/pkg/cataloger/internal/pkgtest"
7	+	)
8	+
9	+	func TestCataloger_Globs(t *testing.T) {
10	+	tests := []struct {
11	+	name string
12	+	fixture string
13	+	expected []string
14	+	}{
15	+	{
16	+	name: "obtain pubspec files",
17	+	fixture: "test-fixtures/glob-paths",
18	+	expected: []string{
19	+	"src/pubspec.lock",
20	+	},
21	+	},
22	+	}
23	+
24	+	for _, test := range tests {
25	+	t.Run(test.name, func(t *testing.T) {
26	+	pkgtest.NewCatalogTester().
27	+	FromDirectory(t, test.fixture).
28	+	ExpectsResolverContentQueries(test.expected).
29	+	TestCataloger(t, NewPubspecLockCataloger())
30	+	})
31	+	}
32	+	}
33	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/dart/test-fixtures/glob-paths/src/pubspec.lock

1

+

bogus pubspec.lock

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/deb/cataloger.go

		skipped 3 lines
4	4		package deb
5	5
6	6		import (
7		-	"github.com/anchore/syft/syft/pkg"
8	7		"github.com/anchore/syft/syft/pkg/cataloger/generic"
9	8		)
10	9
		skipped 2 lines
13	12		// NewDpkgdbCataloger returns a new Deb package cataloger capable of parsing DPKG status DB files.
14	13		func NewDpkgdbCataloger() *generic.Cataloger {
15	14		return generic.NewCataloger(catalogerName).
16		-	WithParserByGlobs(parseDpkgDB, pkg.DpkgDBGlob)
	15	+	// note: these globs have been intentionally split up in order to improve search performance,
	16	+	// please do NOT combine into: "*/var/lib/dpkg/{status,status.d/}"
	17	+	WithParserByGlobs(parseDpkgDB, "/var/lib/dpkg/status", "/var/lib/dpkg/status.d/*")
17	18		}
18	19

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/deb/cataloger_test.go

		skipped 81 lines
82	82		TestCataloger(t, c)
83	83		}
84	84
	85	+	func TestCataloger_Globs(t *testing.T) {
	86	+	tests := []struct {
	87	+	name string
	88	+	fixture string
	89	+	expected []string
	90	+	}{
	91	+	{
	92	+	name: "obtain db status files",
	93	+	fixture: "test-fixtures/glob-paths",
	94	+	expected: []string{
	95	+	"var/lib/dpkg/status",
	96	+	"var/lib/dpkg/status.d/pkg-1.0",
	97	+	},
	98	+	},
	99	+	}
	100	+
	101	+	for _, test := range tests {
	102	+	t.Run(test.name, func(t *testing.T) {
	103	+	pkgtest.NewCatalogTester().
	104	+	FromDirectory(t, test.fixture).
	105	+	ExpectsResolverContentQueries(test.expected).
	106	+	TestCataloger(t, NewDpkgdbCataloger())
	107	+	})
	108	+	}
	109	+	}
	110	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/deb/package.go

		skipped 82 lines
83	83		func addLicenses(resolver source.FileResolver, dbLocation source.Location, p *pkg.Package) {
84	84		metadata, ok := p.Metadata.(pkg.DpkgMetadata)
85	85		if !ok {
86		-	log.WithFields("package", p.String()).Warn("unable to extract DPKG metadata to add licenses")
	86	+	log.WithFields("package", p).Warn("unable to extract DPKG metadata to add licenses")
87	87		return
88	88		}
89	89
		skipped 13 lines
103	103		func mergeFileListing(resolver source.FileResolver, dbLocation source.Location, p *pkg.Package) {
104	104		metadata, ok := p.Metadata.(pkg.DpkgMetadata)
105	105		if !ok {
106		-	log.WithFields("package", p.String()).Warn("unable to extract DPKG metadata to file listing")
	106	+	log.WithFields("package", p).Warn("unable to extract DPKG metadata to file listing")
107	107		return
108	108		}
109	109
		skipped 147 lines

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/deb/test-fixtures/glob-paths/var/lib/dpkg/status

1

+

bogus status

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/deb/test-fixtures/glob-paths/var/lib/dpkg/status.d/pkg-1.0

1

+

bogus package

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/dotnet/cataloger_test.go

1	+	package dotnet
2	+
3	+	import (
4	+	"testing"
5	+
6	+	"github.com/anchore/syft/syft/pkg/cataloger/internal/pkgtest"
7	+	)
8	+
9	+	func TestCataloger_Globs(t *testing.T) {
10	+	tests := []struct {
11	+	name string
12	+	fixture string
13	+	expected []string
14	+	}{
15	+	{
16	+	name: "obtain deps.json files",
17	+	fixture: "test-fixtures/glob-paths",
18	+	expected: []string{
19	+	"src/something.deps.json",
20	+	},
21	+	},
22	+	}
23	+
24	+	for _, test := range tests {
25	+	t.Run(test.name, func(t *testing.T) {
26	+	pkgtest.NewCatalogTester().
27	+	FromDirectory(t, test.fixture).
28	+	ExpectsResolverContentQueries(test.expected).
29	+	TestCataloger(t, NewDotnetDepsCataloger())
30	+	})
31	+	}
32	+	}
33	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/dotnet/test-fixtures/glob-paths/src/something.deps.json

1

+

bogus deps.json

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/elixir/cataloger_test.go

1	+	package elixir
2	+
3	+	import (
4	+	"testing"
5	+
6	+	"github.com/anchore/syft/syft/pkg/cataloger/internal/pkgtest"
7	+	)
8	+
9	+	func TestCataloger_Globs(t *testing.T) {
10	+	tests := []struct {
11	+	name string
12	+	fixture string
13	+	expected []string
14	+	}{
15	+	{
16	+	name: "obtain mix.lock files",
17	+	fixture: "test-fixtures/glob-paths",
18	+	expected: []string{
19	+	"src/mix.lock",
20	+	},
21	+	},
22	+	}
23	+
24	+	for _, test := range tests {
25	+	t.Run(test.name, func(t *testing.T) {
26	+	pkgtest.NewCatalogTester().
27	+	FromDirectory(t, test.fixture).
28	+	ExpectsResolverContentQueries(test.expected).
29	+	TestCataloger(t, NewMixLockCataloger())
30	+	})
31	+	}
32	+	}
33	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/elixir/test-fixtures/glob-paths/src/mix.lock

1

+

bogus mix.lock

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/erlang/cataloger_test.go

1	+	package erlang
2	+
3	+	import (
4	+	"testing"
5	+
6	+	"github.com/anchore/syft/syft/pkg/cataloger/internal/pkgtest"
7	+	)
8	+
9	+	func TestCataloger_Globs(t *testing.T) {
10	+	tests := []struct {
11	+	name string
12	+	fixture string
13	+	expected []string
14	+	}{
15	+	{
16	+	name: "obtain rebar.lock files",
17	+	fixture: "test-fixtures/glob-paths",
18	+	expected: []string{
19	+	"src/rebar.lock",
20	+	},
21	+	},
22	+	}
23	+
24	+	for _, test := range tests {
25	+	t.Run(test.name, func(t *testing.T) {
26	+	pkgtest.NewCatalogTester().
27	+	FromDirectory(t, test.fixture).
28	+	ExpectsResolverContentQueries(test.expected).
29	+	TestCataloger(t, NewRebarLockCataloger())
30	+	})
31	+	}
32	+	}
33	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/erlang/test-fixtures/glob-paths/src/rebar.lock

1

+

bogus rebar.lock

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/generic/cataloger.go

		skipped 46 lines
47	47		c.processor = append(c.processor,
48	48		func(resolver source.FileResolver, env Environment) []request {
49	49		var requests []request
50		-	for _, t := range types {
51		-	log.WithFields("mimetype", t).Trace("searching for paths matching mimetype")
52		-
53		-	matches, err := resolver.FilesByMIMEType(t)
54		-	if err != nil {
55		-	log.Warnf("unable to process mimetype=%q: %+v", t, err)
56		-	continue
57		-	}
58		-	requests = append(requests, makeRequests(parser, matches)...)
	50	+	log.WithFields("mimetypes", types).Trace("searching for paths matching mimetype")
	51	+	matches, err := resolver.FilesByMIMEType(types...)
	52	+	if err != nil {
	53	+	log.Warnf("unable to process mimetypes=%+v: %+v", types, err)
	54	+	return nil
59	55		}
	56	+	requests = append(requests, makeRequests(parser, matches)...)
60	57		return requests
61	58		},
62	59		)
		skipped 95 lines

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/golang/cataloger_test.go

1	+	package golang
2	+
3	+	import (
4	+	"testing"
5	+
6	+	"github.com/anchore/syft/syft/pkg/cataloger/internal/pkgtest"
7	+	)
8	+
9	+	func Test_Mod_Cataloger_Globs(t *testing.T) {
10	+	tests := []struct {
11	+	name string
12	+	fixture string
13	+	expected []string
14	+	}{
15	+	{
16	+	name: "obtain go.mod files",
17	+	fixture: "test-fixtures/glob-paths",
18	+	expected: []string{
19	+	"src/go.mod",
20	+	},
21	+	},
22	+	}
23	+
24	+	for _, test := range tests {
25	+	t.Run(test.name, func(t *testing.T) {
26	+	pkgtest.NewCatalogTester().
27	+	FromDirectory(t, test.fixture).
28	+	ExpectsResolverContentQueries(test.expected).
29	+	IgnoreUnfulfilledPathResponses("src/go.sum").
30	+	TestCataloger(t, NewGoModFileCataloger())
31	+	})
32	+	}
33	+	}
34	+
35	+	func Test_Binary_Cataloger_Globs(t *testing.T) {
36	+	tests := []struct {
37	+	name string
38	+	fixture string
39	+	expected []string
40	+	}{
41	+	{
42	+	name: "obtain binary files",
43	+	fixture: "test-fixtures/glob-paths",
44	+	expected: []string{
45	+	"partial-binary",
46	+	},
47	+	},
48	+	}
49	+
50	+	for _, test := range tests {
51	+	t.Run(test.name, func(t *testing.T) {
52	+	pkgtest.NewCatalogTester().
53	+	FromDirectory(t, test.fixture).
54	+	ExpectsResolverContentQueries(test.expected).
55	+	TestCataloger(t, NewGoModuleBinaryCataloger())
56	+	})
57	+	}
58	+	}
59	+

syft/pkg/cataloger/golang/test-fixtures/glob-paths/partial-binary

Binary file.

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/golang/test-fixtures/glob-paths/src/go.mod

1

+

// bogus go.mod

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/haskell/cataloger_test.go

1	+	package haskell
2	+
3	+	import (
4	+	"testing"
5	+
6	+	"github.com/anchore/syft/syft/pkg/cataloger/internal/pkgtest"
7	+	)
8	+
9	+	func TestCataloger_Globs(t *testing.T) {
10	+	tests := []struct {
11	+	name string
12	+	fixture string
13	+	expected []string
14	+	}{
15	+	{
16	+	name: "obtain stack and cabal files",
17	+	fixture: "test-fixtures/glob-paths",
18	+	expected: []string{
19	+	"src/stack.yaml",
20	+	"src/stack.yaml.lock",
21	+	"src/cabal.project.freeze",
22	+	},
23	+	},
24	+	}
25	+
26	+	for _, test := range tests {
27	+	t.Run(test.name, func(t *testing.T) {
28	+	pkgtest.NewCatalogTester().
29	+	FromDirectory(t, test.fixture).
30	+	ExpectsResolverContentQueries(test.expected).
31	+	TestCataloger(t, NewHackageCataloger())
32	+	})
33	+	}
34	+	}
35	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/haskell/test-fixtures/glob-paths/src/cabal.project.freeze

1

+

cabal.project.freeze

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/haskell/test-fixtures/glob-paths/src/stack.yaml

1

+

bogus stack.yaml

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/haskell/test-fixtures/glob-paths/src/stack.yaml.lock

1

+

bogus stack.yaml.lock

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/internal/pkgtest/observing_resolver.go

1	+	package pkgtest
2	+
3	+	import (
4	+	"fmt"
5	+	"io"
6	+	"sort"
7	+
8	+	"github.com/scylladb/go-set/strset"
9	+
10	+	"github.com/anchore/syft/syft/source"
11	+	)
12	+
13	+	var _ source.FileResolver = (*ObservingResolver)(nil)
14	+
15	+	type ObservingResolver struct {
16	+	decorated source.FileResolver
17	+	pathQueries map[string][]string
18	+	pathResponses []source.Location
19	+	contentQueries []source.Location
20	+	emptyPathResponses map[string][]string
21	+	}
22	+
23	+	func NewObservingResolver(resolver source.FileResolver) *ObservingResolver {
24	+	return &ObservingResolver{
25	+	decorated: resolver,
26	+	pathResponses: make([]source.Location, 0),
27	+	emptyPathResponses: make(map[string][]string),
28	+	pathQueries: make(map[string][]string),
29	+	}
30	+	}
31	+
32	+	// testing helpers...
33	+
34	+	func (r *ObservingResolver) ObservedPathQuery(input string) bool {
35	+	for _, vs := range r.pathQueries {
36	+	for _, v := range vs {
37	+	if v == input {
38	+	return true
39	+	}
40	+	}
41	+	}
42	+	return false
43	+	}
44	+
45	+	func (r *ObservingResolver) ObservedPathResponses(path string) bool {
46	+	for _, loc := range r.pathResponses {
47	+	if loc.RealPath == path {
48	+	return true
49	+	}
50	+	}
51	+	return false
52	+	}
53	+
54	+	func (r *ObservingResolver) ObservedContentQueries(path string) bool {
55	+	for _, loc := range r.contentQueries {
56	+	if loc.RealPath == path {
57	+	return true
58	+	}
59	+	}
60	+	return false
61	+	}
62	+
63	+	func (r *ObservingResolver) AllContentQueries() []string {
64	+	observed := strset.New()
65	+	for _, loc := range r.contentQueries {
66	+	observed.Add(loc.RealPath)
67	+	}
68	+	return observed.List()
69	+	}
70	+
71	+	func (r *ObservingResolver) AllPathQueries() map[string][]string {
72	+	return r.pathQueries
73	+	}
74	+
75	+	func (r *ObservingResolver) PruneUnfulfilledPathResponses(ignore map[string][]string, ignorePaths ...string) {
76	+	if ignore == nil {
77	+	return
78	+	}
79	+	// remove any paths that were ignored for specific calls
80	+	for k, v := range ignore {
81	+	results := r.emptyPathResponses[k]
82	+	for _, ig := range v {
83	+	for i, result := range results {
84	+	if result == ig {
85	+	results = append(results[:i], results[i+1:]...)
86	+	break
87	+	}
88	+	}
89	+	}
90	+	if len(results) > 0 {
91	+	r.emptyPathResponses[k] = results
92	+	} else {
93	+	delete(r.emptyPathResponses, k)
94	+	}
95	+	}
96	+
97	+	// remove any paths that were ignored for all calls
98	+	for _, ig := range ignorePaths {
99	+	for k, v := range r.emptyPathResponses {
100	+	for i, result := range v {
101	+	if result == ig {
102	+	v = append(v[:i], v[i+1:]...)
103	+	break
104	+	}
105	+	}
106	+	if len(v) > 0 {
107	+	r.emptyPathResponses[k] = v
108	+	} else {
109	+	delete(r.emptyPathResponses, k)
110	+	}
111	+	}
112	+	}
113	+	}
114	+
115	+	func (r *ObservingResolver) HasUnfulfilledPathRequests() bool {
116	+	return len(r.emptyPathResponses) > 0
117	+	}
118	+
119	+	func (r *ObservingResolver) PrettyUnfulfilledPathRequests() string {
120	+	var res string
121	+	var keys []string
122	+
123	+	for k := range r.emptyPathResponses {
124	+	keys = append(keys, k)
125	+	}
126	+
127	+	sort.Strings(keys)
128	+
129	+	for _, k := range keys {
130	+	res += fmt.Sprintf(" %s: %+v\n", k, r.emptyPathResponses[k])
131	+	}
132	+	return res
133	+	}
134	+
135	+	// For the file path resolver...
136	+
137	+	func (r *ObservingResolver) addPathQuery(name string, input ...string) {
138	+	r.pathQueries[name] = append(r.pathQueries[name], input...)
139	+	}
140	+
141	+	func (r *ObservingResolver) addPathResponse(locs ...source.Location) {
142	+	r.pathResponses = append(r.pathResponses, locs...)
143	+	}
144	+
145	+	func (r *ObservingResolver) addEmptyPathResponse(name string, locs []source.Location, paths ...string) {
146	+	if len(locs) == 0 {
147	+	results := r.emptyPathResponses[name]
148	+	results = append(results, paths...)
149	+	r.emptyPathResponses[name] = results
150	+	}
151	+	}
152	+
153	+	func (r *ObservingResolver) FilesByPath(paths ...string) ([]source.Location, error) {
154	+	name := "FilesByPath"
155	+	r.addPathQuery(name, paths...)
156	+
157	+	locs, err := r.decorated.FilesByPath(paths...)
158	+
159	+	r.addPathResponse(locs...)
160	+	r.addEmptyPathResponse(name, locs, paths...)
161	+	return locs, err
162	+	}
163	+
164	+	func (r *ObservingResolver) FilesByGlob(patterns ...string) ([]source.Location, error) {
165	+	name := "FilesByGlob"
166	+	r.addPathQuery(name, patterns...)
167	+
168	+	locs, err := r.decorated.FilesByGlob(patterns...)
169	+
170	+	r.addPathResponse(locs...)
171	+	r.addEmptyPathResponse(name, locs, patterns...)
172	+	return locs, err
173	+	}
174	+
175	+	func (r *ObservingResolver) FilesByMIMEType(types ...string) ([]source.Location, error) {
176	+	name := "FilesByMIMEType"
177	+	r.addPathQuery(name, types...)
178	+
179	+	locs, err := r.decorated.FilesByMIMEType(types...)
180	+
181	+	r.addPathResponse(locs...)
182	+	r.addEmptyPathResponse(name, locs, types...)
183	+	return locs, err
184	+	}
185	+
186	+	func (r ObservingResolver) RelativeFileByPath(l source.Location, path string) source.Location {
187	+	name := "RelativeFileByPath"
188	+	r.addPathQuery(name, path)
189	+
190	+	loc := r.decorated.RelativeFileByPath(l, path)
191	+
192	+	if loc != nil {
193	+	r.addPathResponse(*loc)
194	+	} else {
195	+	results := r.emptyPathResponses[name]
196	+	results = append(results, path)
197	+	r.emptyPathResponses[name] = results
198	+	}
199	+	return loc
200	+	}
201	+
202	+	// For the content resolver methods...
203	+
204	+	func (r *ObservingResolver) FileContentsByLocation(location source.Location) (io.ReadCloser, error) {
205	+	r.contentQueries = append(r.contentQueries, location)
206	+	reader, err := r.decorated.FileContentsByLocation(location)
207	+	return reader, err
208	+	}
209	+
210	+	// For the remaining resolver methods...
211	+
212	+	func (r *ObservingResolver) AllLocations() <-chan source.Location {
213	+	return r.decorated.AllLocations()
214	+	}
215	+
216	+	func (r *ObservingResolver) HasPath(s string) bool {
217	+	return r.decorated.HasPath(s)
218	+	}
219	+
220	+	func (r *ObservingResolver) FileMetadataByLocation(location source.Location) (source.FileMetadata, error) {
221	+	return r.decorated.FileMetadataByLocation(location)
222	+	}
223	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/internal/pkgtest/test_generic_parser.go

1	1		package pkgtest
2	2
3	3		import (
	4	+	"fmt"
4	5		"io"
5	6		"os"
6	7		"strings"
		skipped 1 lines
8	9
9	10		"github.com/google/go-cmp/cmp"
10	11		"github.com/google/go-cmp/cmp/cmpopts"
	12	+	"github.com/stretchr/testify/assert"
11	13		"github.com/stretchr/testify/require"
12	14
13	15		"github.com/anchore/stereoscope/pkg/imagetest"
		skipped 7 lines
21	23		type locationComparer func(x, y source.Location) bool
22	24
23	25		type CatalogTester struct {
24		-	expectedPkgs []pkg.Package
25		-	expectedRelationships []artifact.Relationship
26		-	env *generic.Environment
27		-	reader source.LocationReadCloser
28		-	resolver source.FileResolver
29		-	wantErr require.ErrorAssertionFunc
30		-	compareOptions []cmp.Option
31		-	locationComparer locationComparer
	26	+	expectedPkgs []pkg.Package
	27	+	expectedRelationships []artifact.Relationship
	28	+	assertResultExpectations bool
	29	+	expectedPathResponses []string // this is a minimum set, the resolver may return more that just this list
	30	+	expectedContentQueries []string // this is a full set, any other queries are unexpected (and will fail the test)
	31	+	ignoreUnfulfilledPathResponses map[string][]string
	32	+	ignoreAnyUnfulfilledPaths []string
	33	+	env *generic.Environment
	34	+	reader source.LocationReadCloser
	35	+	resolver source.FileResolver
	36	+	wantErr require.ErrorAssertionFunc
	37	+	compareOptions []cmp.Option
	38	+	locationComparer locationComparer
32	39		}
33	40
34	41		func NewCatalogTester() *CatalogTester {
35	42		return &CatalogTester{
36	43		wantErr: require.NoError,
37	44		locationComparer: DefaultLocationComparer,
	45	+	ignoreUnfulfilledPathResponses: map[string][]string{
	46	+	"FilesByPath": {
	47	+	// most catalogers search for a linux release, which will not be fulfilled in testing
	48	+	"/etc/os-release",
	49	+	"/usr/lib/os-release",
	50	+	"/etc/system-release-cpe",
	51	+	"/etc/redhat-release",
	52	+	"/bin/busybox",
	53	+	},
	54	+	},
38	55		}
39	56		}
40	57
		skipped 49 lines
90	107		}
91	108
92	109		func (p CatalogTester) WithError() CatalogTester {
	110	+	p.assertResultExpectations = true
93	111		p.wantErr = require.Error
94	112		return p
95	113		}
		skipped 33 lines
129	147		return p
130	148		}
131	149
	150	+	func (p CatalogTester) WithCompareOptions(opts ...cmp.Option) CatalogTester {
	151	+	p.compareOptions = append(p.compareOptions, opts...)
	152	+	return p
	153	+	}
	154	+
132	155		func (p CatalogTester) Expects(pkgs []pkg.Package, relationships []artifact.Relationship) CatalogTester {
	156	+	p.assertResultExpectations = true
133	157		p.expectedPkgs = pkgs
134	158		p.expectedRelationships = relationships
135	159		return p
136	160		}
137	161
	162	+	func (p CatalogTester) ExpectsResolverPathResponses(locations []string) CatalogTester {
	163	+	p.expectedPathResponses = locations
	164	+	return p
	165	+	}
	166	+
	167	+	func (p CatalogTester) ExpectsResolverContentQueries(locations []string) CatalogTester {
	168	+	p.expectedContentQueries = locations
	169	+	return p
	170	+	}
	171	+
	172	+	func (p CatalogTester) IgnoreUnfulfilledPathResponses(paths ...string) CatalogTester {
	173	+	p.ignoreAnyUnfulfilledPaths = append(p.ignoreAnyUnfulfilledPaths, paths...)
	174	+	return p
	175	+	}
	176	+
138	177		func (p CatalogTester) TestParser(t testing.T, parser generic.Parser) {
139	178		t.Helper()
140	179		pkgs, relationships, err := parser(p.resolver, p.env, p.reader)
		skipped 3 lines
144	183
145	184		func (p CatalogTester) TestCataloger(t testing.T, cataloger pkg.Cataloger) {
146	185		t.Helper()
147		-	pkgs, relationships, err := cataloger.Catalog(p.resolver)
148		-	p.wantErr(t, err)
149		-	p.assertPkgs(t, pkgs, relationships)
	186	+
	187	+	resolver := NewObservingResolver(p.resolver)
	188	+
	189	+	pkgs, relationships, err := cataloger.Catalog(resolver)
	190	+
	191	+	// this is a minimum set, the resolver may return more that just this list
	192	+	for _, path := range p.expectedPathResponses {
	193	+	assert.Truef(t, resolver.ObservedPathResponses(path), "expected path query for %q was not observed", path)
	194	+	}
	195	+
	196	+	// this is a full set, any other queries are unexpected (and will fail the test)
	197	+	if len(p.expectedContentQueries) > 0 {
	198	+	assert.ElementsMatchf(t, p.expectedContentQueries, resolver.AllContentQueries(), "unexpected content queries observed: diff %s", cmp.Diff(p.expectedContentQueries, resolver.AllContentQueries()))
	199	+	}
	200	+
	201	+	if p.assertResultExpectations {
	202	+	p.wantErr(t, err)
	203	+	p.assertPkgs(t, pkgs, relationships)
	204	+	} else {
	205	+	resolver.PruneUnfulfilledPathResponses(p.ignoreUnfulfilledPathResponses, p.ignoreAnyUnfulfilledPaths...)
	206	+
	207	+	// if we aren't testing the results, we should focus on what was searched for (for glob-centric tests)
	208	+	assert.Falsef(t, resolver.HasUnfulfilledPathRequests(), "unfulfilled path requests: \n%v", resolver.PrettyUnfulfilledPathRequests())
	209	+	}
150	210		}
151	211
152	212		func (p CatalogTester) assertPkgs(t testing.T, pkgs []pkg.Package, relationships []artifact.Relationship) {
		skipped 22 lines
175	235		),
176	236		)
177	237
178		-	if diff := cmp.Diff(p.expectedPkgs, pkgs, p.compareOptions...); diff != "" {
179		-	t.Errorf("unexpected packages from parsing (-expected +actual)\n%s", diff)
	238	+	{
	239	+	var r diffReporter
	240	+	var opts []cmp.Option
	241	+
	242	+	opts = append(opts, p.compareOptions...)
	243	+	opts = append(opts, cmp.Reporter(&r))
	244	+
	245	+	if diff := cmp.Diff(p.expectedPkgs, pkgs, opts...); diff != "" {
	246	+	t.Log("Specific Differences:\n" + r.String())
	247	+	t.Errorf("unexpected packages from parsing (-expected +actual)\n%s", diff)
	248	+	}
180	249		}
181	250
182		-	if diff := cmp.Diff(p.expectedRelationships, relationships, p.compareOptions...); diff != "" {
183		-	t.Errorf("unexpected relationships from parsing (-expected +actual)\n%s", diff)
	251	+	{
	252	+	var r diffReporter
	253	+	var opts []cmp.Option
	254	+
	255	+	opts = append(opts, p.compareOptions...)
	256	+	opts = append(opts, cmp.Reporter(&r))
	257	+
	258	+	if diff := cmp.Diff(p.expectedRelationships, relationships, opts...); diff != "" {
	259	+	t.Log("Specific Differences:\n" + r.String())
	260	+
	261	+	t.Errorf("unexpected relationships from parsing (-expected +actual)\n%s", diff)
	262	+	}
184	263		}
185	264		}
186	265
		skipped 37 lines
224	303		}
225	304		}
226	305
	306	+	// diffReporter is a simple custom reporter that only records differences detected during comparison.
	307	+	type diffReporter struct {
	308	+	path cmp.Path
	309	+	diffs []string
	310	+	}
	311	+
	312	+	func (r *diffReporter) PushStep(ps cmp.PathStep) {
	313	+	r.path = append(r.path, ps)
	314	+	}
	315	+
	316	+	func (r *diffReporter) Report(rs cmp.Result) {
	317	+	if !rs.Equal() {
	318	+	vx, vy := r.path.Last().Values()
	319	+	r.diffs = append(r.diffs, fmt.Sprintf("%#v:\n\t-: %+v\n\t+: %+v\n", r.path, vx, vy))
	320	+	}
	321	+	}
	322	+
	323	+	func (r *diffReporter) PopStep() {
	324	+	r.path = r.path[:len(r.path)-1]
	325	+	}
	326	+
	327	+	func (r *diffReporter) String() string {
	328	+	return strings.Join(r.diffs, "\n")
	329	+	}
	330	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/cataloger.go

		skipped 28 lines
29	29		// Pom files list dependencies that maybe not be locally installed yet.
30	30		func NewJavaPomCataloger() *generic.Cataloger {
31	31		return generic.NewCataloger("java-pom-cataloger").
32		-	WithParserByGlobs(parserPomXML, pomXMLDirGlob)
	32	+	WithParserByGlobs(parserPomXML, "**/pom.xml")
33	33		}
34	34

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/cataloger_test.go

1	+	package java
2	+
3	+	import (
4	+	"testing"
5	+
6	+	"github.com/anchore/syft/syft/pkg/cataloger/internal/pkgtest"
7	+	)
8	+
9	+	func Test_ArchiveCataloger_Globs(t *testing.T) {
10	+	tests := []struct {
11	+	name string
12	+	fixture string
13	+	expected []string
14	+	}{
15	+	{
16	+	name: "obtain java archive files",
17	+	fixture: "test-fixtures/glob-paths",
18	+	expected: []string{
19	+	"java-archives/example.jar",
20	+	"java-archives/example.war",
21	+	"java-archives/example.ear",
22	+	"java-archives/example.par",
23	+	"java-archives/example.sar",
24	+	"java-archives/example.jpi",
25	+	"java-archives/example.hpi",
26	+	"java-archives/example.lpkg",
27	+	"archives/example.zip",
28	+	"archives/example.tar",
29	+	"archives/example.tar.gz",
30	+	"archives/example.tgz",
31	+	"archives/example.tar.bz",
32	+	"archives/example.tar.bz2",
33	+	"archives/example.tbz",
34	+	"archives/example.tbz2",
35	+	"archives/example.tar.br",
36	+	"archives/example.tbr",
37	+	"archives/example.tar.lz4",
38	+	"archives/example.tlz4",
39	+	"archives/example.tar.sz",
40	+	"archives/example.tsz",
41	+	"archives/example.tar.xz",
42	+	"archives/example.txz",
43	+	"archives/example.tar.zst",
44	+	"archives/example.tzst",
45	+	"archives/example.tar.zstd",
46	+	"archives/example.tzstd",
47	+	},
48	+	},
49	+	}
50	+
51	+	for _, test := range tests {
52	+	t.Run(test.name, func(t *testing.T) {
53	+	pkgtest.NewCatalogTester().
54	+	FromDirectory(t, test.fixture).
55	+	ExpectsResolverContentQueries(test.expected).
56	+	TestCataloger(t, NewJavaCataloger(Config{
57	+	SearchUnindexedArchives: true,
58	+	SearchIndexedArchives: true,
59	+	}))
60	+	})
61	+	}
62	+	}
63	+
64	+	func Test_POMCataloger_Globs(t *testing.T) {
65	+	tests := []struct {
66	+	name string
67	+	fixture string
68	+	expected []string
69	+	}{
70	+	{
71	+	name: "obtain java pom files",
72	+	fixture: "test-fixtures/glob-paths",
73	+	expected: []string{
74	+	"src/pom.xml",
75	+	},
76	+	},
77	+	}
78	+
79	+	for _, test := range tests {
80	+	t.Run(test.name, func(t *testing.T) {
81	+	pkgtest.NewCatalogTester().
82	+	FromDirectory(t, test.fixture).
83	+	ExpectsResolverContentQueries(test.expected).
84	+	TestCataloger(t, NewJavaPomCataloger())
85	+	})
86	+	}
87	+	}
88	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/parse_pom_xml.go

		skipped 17 lines
18	18		)
19	19
20	20		const pomXMLGlob = "*pom.xml"
21		-	const pomXMLDirGlob = "**/pom.xml"
22	21
23	22		var propertyMatcher = regexp.MustCompile("[$][{][^}]+[}]")
24	23
		skipped 139 lines

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/tar_wrapped_archive_parser.go

		skipped 33 lines
34	34		"*/.txz",
35	35		// zst
36	36		"*/.tar.zst",
	37	+	"*/.tzst",
	38	+	"*/.tar.zstd",
	39	+	"*/.tzstd",
37	40		}
38	41
39	42		// TODO: when the generic archive cataloger is implemented, this should be removed (https://github.com/anchore/syft/issues/246)
		skipped 26 lines

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/archives/.gitignore

	1	+	# we want to override some of the root level ignores just for the fixutes that we know are safe
	2	+	!*

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/archives/example.tar

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/archives/example.tar.br

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/archives/example.tar.bz

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/archives/example.tar.bz2

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/archives/example.tar.gz

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/archives/example.tar.lz4

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/archives/example.tar.sz

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/archives/example.tar.xz

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/archives/example.tar.zst

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/archives/example.tar.zstd

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/archives/example.tbr

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/archives/example.tbz

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/archives/example.tbz2

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/archives/example.tgz

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/archives/example.tlz4

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/archives/example.tsz

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/archives/example.txz

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/archives/example.tzst

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/archives/example.tzstd

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/archives/example.zip

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/java-archives/.gitignore

	1	+	# we want to override some of the root level ignores just for the fixutes that we know are safe
	2	+	!*

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/java-archives/example.ear

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/java-archives/example.hpi

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/java-archives/example.jar

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/java-archives/example.jpi

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/java-archives/example.lpkg

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/java-archives/example.par

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/java-archives/example.sar

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/java-archives/example.war

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/java/test-fixtures/glob-paths/src/pom.xml

1

+

bogus pom.xml

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/javascript/cataloger.go

		skipped 6 lines
7	7		"github.com/anchore/syft/syft/pkg/cataloger/generic"
8	8		)
9	9
10		-	// NewJavascriptPackageCataloger returns a new JavaScript cataloger object based on detection of npm based packages.
11		-	func NewJavascriptPackageCataloger() *generic.Cataloger {
	10	+	// NewPackageCataloger returns a new JavaScript cataloger object based on detection of npm based packages.
	11	+	func NewPackageCataloger() *generic.Cataloger {
12	12		return generic.NewCataloger("javascript-package-cataloger").
13	13		WithParserByGlobs(parsePackageJSON, "**/package.json")
14	14		}
15	15
16		-	func NewJavascriptLockCataloger() *generic.Cataloger {
	16	+	// NewLockCataloger returns a new JavaScript cataloger object based on detection of lock files.
	17	+	func NewLockCataloger() *generic.Cataloger {
17	18		return generic.NewCataloger("javascript-lock-cataloger").
18	19		WithParserByGlobs(parsePackageLock, "**/package-lock.json").
19	20		WithParserByGlobs(parseYarnLock, "**/yarn.lock").
		skipped 3 lines

■ ■ ■ ■ ■ ■ ■

syft/pkg/cataloger/javascript/cataloger_test.go

		skipped 138 lines
139	139		pkgtest.NewCatalogTester().
140	140		FromDirectory(t, "test-fixtures/pkg-lock").
141	141		Expects(expectedPkgs, nil).
142		-	TestCataloger(t, NewJavascriptLockCataloger())
	142	+	TestCataloger(t, NewLockCataloger())
	143	+
	144	+	}
	145	+
	146	+	func Test_PackageCataloger_Globs(t *testing.T) {
	147	+	tests := []struct {
	148	+	name string
	149	+	fixture string
	150	+	expected []string
	151	+	}{
	152	+	{
	153	+	name: "obtain package files",
	154	+	fixture: "test-fixtures/glob-paths",
	155	+	expected: []string{
	156	+	"src/package.json",
	157	+	},
	158	+	},
	159	+	}
143	160
	161	+	for _, test := range tests {
	162	+	t.Run(test.name, func(t *testing.T) {
	163	+	pkgtest.NewCatalogTester().
	164	+	FromDirectory(t, test.fixture).
	165	+	ExpectsResolverContentQueries(test.expected).
	166	+	TestCataloger(t, NewPackageCataloger())
	167	+	})
	168	+	}
	169	+	}
	170	+
	171	+	func Test_LockCataloger_Globs(t *testing.T) {
	172	+	tests := []struct {
	173	+	name string
	174	+	fixture string
	175	+	expected []string
	176	+	}{
	177	+	{
	178	+	name: "obtain package files",
	179	+	fixture: "test-fixtures/glob-paths",
	180	+	expected: []string{
	181	+	"src/package-lock.json",
	182	+	"src/pnpm-lock.yaml",
	183	+	"src/yarn.lock",
	184	+	},
	185	+	},
	186	+	}
	187	+
	188	+	for _, test := range tests {
	189	+	t.Run(test.name, func(t *testing.T) {
	190	+	pkgtest.NewCatalogTester().
	191	+	FromDirectory(t, test.fixture).
	192	+	ExpectsResolverContentQueries(test.expected).
	193	+	TestCataloger(t, NewLockCataloger())
	194	+	})
	195	+	}
144	196		}
145	197

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/javascript/test-fixtures/glob-paths/src/package-lock.json

1

+

bogus package-lock.json

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/javascript/test-fixtures/glob-paths/src/package.json

1

+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/javascript/test-fixtures/glob-paths/src/pnpm-lock.yaml

1

+

bogus pnpm-lock.yaml

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/javascript/test-fixtures/glob-paths/src/yarn.lock

1

+

bogus yarn.lock

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/php/cataloger.go

		skipped 6 lines
7	7		"github.com/anchore/syft/syft/pkg/cataloger/generic"
8	8		)
9	9
10		-	// NewPHPComposerInstalledCataloger returns a new cataloger for PHP installed.json files.
11		-	func NewPHPComposerInstalledCataloger() *generic.Cataloger {
	10	+	// NewComposerInstalledCataloger returns a new cataloger for PHP installed.json files.
	11	+	func NewComposerInstalledCataloger() *generic.Cataloger {
12	12		return generic.NewCataloger("php-composer-installed-cataloger").
13	13		WithParserByGlobs(parseInstalledJSON, "**/installed.json")
14	14		}
15	15
16		-	// NewPHPComposerLockCataloger returns a new cataloger for PHP composer.lock files.
17		-	func NewPHPComposerLockCataloger() *generic.Cataloger {
	16	+	// NewComposerLockCataloger returns a new cataloger for PHP composer.lock files.
	17	+	func NewComposerLockCataloger() *generic.Cataloger {
18	18		return generic.NewCataloger("php-composer-lock-cataloger").
19	19		WithParserByGlobs(parseComposerLock, "**/composer.lock")
20	20		}
		skipped 1 lines

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/php/cataloger_test.go

1	+	package php
2	+
3	+	import (
4	+	"testing"
5	+
6	+	"github.com/anchore/syft/syft/pkg/cataloger/internal/pkgtest"
7	+	)
8	+
9	+	func Test_ComposerInstalledCataloger_Globs(t *testing.T) {
10	+	tests := []struct {
11	+	name string
12	+	fixture string
13	+	expected []string
14	+	}{
15	+	{
16	+	name: "obtain composer files",
17	+	fixture: "test-fixtures/glob-paths",
18	+	expected: []string{
19	+	"src/installed.json",
20	+	},
21	+	},
22	+	}
23	+
24	+	for _, test := range tests {
25	+	t.Run(test.name, func(t *testing.T) {
26	+	pkgtest.NewCatalogTester().
27	+	FromDirectory(t, test.fixture).
28	+	ExpectsResolverContentQueries(test.expected).
29	+	TestCataloger(t, NewComposerInstalledCataloger())
30	+	})
31	+	}
32	+	}
33	+
34	+	func Test_ComposerLockCataloger_Globs(t *testing.T) {
35	+	tests := []struct {
36	+	name string
37	+	fixture string
38	+	expected []string
39	+	}{
40	+	{
41	+	name: "obtain composer lock files",
42	+	fixture: "test-fixtures/glob-paths",
43	+	expected: []string{
44	+	"src/composer.lock",
45	+	},
46	+	},
47	+	}
48	+
49	+	for _, test := range tests {
50	+	t.Run(test.name, func(t *testing.T) {
51	+	pkgtest.NewCatalogTester().
52	+	FromDirectory(t, test.fixture).
53	+	ExpectsResolverContentQueries(test.expected).
54	+	TestCataloger(t, NewComposerLockCataloger())
55	+	})
56	+	}
57	+	}
58	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/php/test-fixtures/glob-paths/src/composer.lock

1

+

bogus composer.lock

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/php/test-fixtures/glob-paths/src/installed.json

1

+

bogus installed.json

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/portage/cataloger.go

		skipped 3 lines
4	4		package portage
5	5
6	6		import (
7		-	"bufio"
8		-	"fmt"
9		-	"path"
10		-	"path/filepath"
11		-	"regexp"
12		-	"sort"
13		-	"strconv"
14		-	"strings"
15		-
16		-	"github.com/anchore/syft/internal"
17		-	"github.com/anchore/syft/internal/log"
18		-	"github.com/anchore/syft/syft/artifact"
19		-	"github.com/anchore/syft/syft/file"
20		-	"github.com/anchore/syft/syft/pkg"
21	7		"github.com/anchore/syft/syft/pkg/cataloger/generic"
22		-	"github.com/anchore/syft/syft/source"
23		-	)
24		-
25		-	var (
26		-	cpvRe = regexp.MustCompile(`/([^/]/[\w+][\w+-])-((\d+)((\.\d+))([a-z]?)((_(pre\|p\|beta\|alpha\|rc)\d)*)(-r\d+)?)/CONTENTS$`)
27		-	_ generic.Parser = parsePortageContents
28	8		)
29	9
30	10		func NewPortageCataloger() *generic.Cataloger {
		skipped 1 lines
32	12		WithParserByGlobs(parsePortageContents, "*/var/db/pkg//*/CONTENTS")
33	13		}
34	14
35		-	func parsePortageContents(resolver source.FileResolver, _ *generic.Environment, reader source.LocationReadCloser) ([]pkg.Package, []artifact.Relationship, error) {
36		-	cpvMatch := cpvRe.FindStringSubmatch(reader.Location.RealPath)
37		-	if cpvMatch == nil {
38		-	return nil, nil, fmt.Errorf("failed to match package and version in %s", reader.Location.RealPath)
39		-	}
40		-
41		-	name, version := cpvMatch[1], cpvMatch[2]
42		-	if name == "" \|\| version == "" {
43		-	log.WithFields("path", reader.Location.RealPath).Warnf("failed to parse portage name and version")
44		-	return nil, nil, nil
45		-	}
46		-
47		-	p := pkg.Package{
48		-	Name: name,
49		-	Version: version,
50		-	PURL: packageURL(name, version),
51		-	Locations: source.NewLocationSet(),
52		-	Type: pkg.PortagePkg,
53		-	MetadataType: pkg.PortageMetadataType,
54		-	Metadata: pkg.PortageMetadata{
55		-	// ensure the default value for a collection is never nil since this may be shown as JSON
56		-	Files: make([]pkg.PortageFileRecord, 0),
57		-	},
58		-	}
59		-	addLicenses(resolver, reader.Location, &p)
60		-	addSize(resolver, reader.Location, &p)
61		-	addFiles(resolver, reader.Location, &p)
62		-
63		-	p.SetID()
64		-
65		-	return []pkg.Package{p}, nil, nil
66		-	}
67		-
68		-	func addFiles(resolver source.FileResolver, dbLocation source.Location, p *pkg.Package) {
69		-	contentsReader, err := resolver.FileContentsByLocation(dbLocation)
70		-	if err != nil {
71		-	log.WithFields("path", dbLocation.RealPath).Warnf("failed to fetch portage contents (package=%s): %+v", p.Name, err)
72		-	return
73		-	}
74		-
75		-	entry, ok := p.Metadata.(pkg.PortageMetadata)
76		-	if !ok {
77		-	return
78		-	}
79		-
80		-	scanner := bufio.NewScanner(contentsReader)
81		-	for scanner.Scan() {
82		-	line := strings.Trim(scanner.Text(), "\n")
83		-	fields := strings.Split(line, " ")
84		-
85		-	if fields[0] == "obj" {
86		-	record := pkg.PortageFileRecord{
87		-	Path: fields[1],
88		-	}
89		-	record.Digest = &file.Digest{
90		-	Algorithm: "md5",
91		-	Value: fields[2],
92		-	}
93		-	entry.Files = append(entry.Files, record)
94		-	}
95		-	}
96		-
97		-	p.Metadata = entry
98		-	p.Locations.Add(dbLocation)
99		-	}
100		-
101		-	func addLicenses(resolver source.FileResolver, dbLocation source.Location, p *pkg.Package) {
102		-	parentPath := filepath.Dir(dbLocation.RealPath)
103		-
104		-	location := resolver.RelativeFileByPath(dbLocation, path.Join(parentPath, "LICENSE"))
105		-
106		-	if location == nil {
107		-	return
108		-	}
109		-
110		-	licenseReader, err := resolver.FileContentsByLocation(*location)
111		-	if err != nil {
112		-	log.WithFields("path", dbLocation.RealPath).Warnf("failed to fetch portage LICENSE: %+v", err)
113		-	return
114		-	}
115		-
116		-	findings := internal.NewStringSet()
117		-	scanner := bufio.NewScanner(licenseReader)
118		-	scanner.Split(bufio.ScanWords)
119		-	for scanner.Scan() {
120		-	token := scanner.Text()
121		-	if token != "\|\|" && token != "(" && token != ")" {
122		-	findings.Add(token)
123		-	}
124		-	}
125		-	licenses := findings.ToSlice()
126		-	sort.Strings(licenses)
127		-	p.Licenses = licenses
128		-	p.Locations.Add(*location)
129		-	}
130		-
131		-	func addSize(resolver source.FileResolver, dbLocation source.Location, p *pkg.Package) {
132		-	parentPath := filepath.Dir(dbLocation.RealPath)
133		-
134		-	location := resolver.RelativeFileByPath(dbLocation, path.Join(parentPath, "SIZE"))
135		-
136		-	if location == nil {
137		-	return
138		-	}
139		-
140		-	entry, ok := p.Metadata.(pkg.PortageMetadata)
141		-	if !ok {
142		-	return
143		-	}
144		-
145		-	sizeReader, err := resolver.FileContentsByLocation(*location)
146		-	if err != nil {
147		-	log.WithFields("name", p.Name).Warnf("failed to fetch portage SIZE: %+v", err)
148		-	return
149		-	}
150		-
151		-	scanner := bufio.NewScanner(sizeReader)
152		-	for scanner.Scan() {
153		-	line := strings.Trim(scanner.Text(), "\n")
154		-	size, err := strconv.Atoi(line)
155		-	if err == nil {
156		-	entry.InstalledSize = size
157		-	}
158		-	}
159		-
160		-	p.Metadata = entry
161		-	p.Locations.Add(*location)
162		-	}
163		-

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/portage/cataloger_test.go

		skipped 71 lines
72	72
73	73		}
74	74
	75	+	func TestCataloger_Globs(t *testing.T) {
	76	+	tests := []struct {
	77	+	name string
	78	+	fixture string
	79	+	expected []string
	80	+	}{
	81	+	{
	82	+	name: "obtain portage contents file",
	83	+	fixture: "test-fixtures/glob-paths",
	84	+	expected: []string{
	85	+	"var/db/pkg/x/y/CONTENTS",
	86	+	},
	87	+	},
	88	+	}
	89	+
	90	+	for _, test := range tests {
	91	+	t.Run(test.name, func(t *testing.T) {
	92	+	pkgtest.NewCatalogTester().
	93	+	FromDirectory(t, test.fixture).
	94	+	ExpectsResolverContentQueries(test.expected).
	95	+	TestCataloger(t, NewPortageCataloger())
	96	+	})
	97	+	}
	98	+	}
	99	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/portage/parse_portage_contents.go

1	+	package portage
2	+
3	+	import (
4	+	"bufio"
5	+	"fmt"
6	+	"path"
7	+	"path/filepath"
8	+	"regexp"
9	+	"sort"
10	+	"strconv"
11	+	"strings"
12	+
13	+	"github.com/anchore/syft/internal"
14	+	"github.com/anchore/syft/internal/log"
15	+	"github.com/anchore/syft/syft/artifact"
16	+	"github.com/anchore/syft/syft/file"
17	+	"github.com/anchore/syft/syft/pkg"
18	+	"github.com/anchore/syft/syft/pkg/cataloger/generic"
19	+	"github.com/anchore/syft/syft/source"
20	+	)
21	+
22	+	var (
23	+	cpvRe = regexp.MustCompile(`/([^/]/[\w+][\w+-])-((\d+)((\.\d+))([a-z]?)((_(pre\|p\|beta\|alpha\|rc)\d)*)(-r\d+)?)/CONTENTS$`)
24	+	_ generic.Parser = parsePortageContents
25	+	)
26	+
27	+	func parsePortageContents(resolver source.FileResolver, _ *generic.Environment, reader source.LocationReadCloser) ([]pkg.Package, []artifact.Relationship, error) {
28	+	cpvMatch := cpvRe.FindStringSubmatch(reader.Location.RealPath)
29	+	if cpvMatch == nil {
30	+	return nil, nil, fmt.Errorf("failed to match package and version in %s", reader.Location.RealPath)
31	+	}
32	+
33	+	name, version := cpvMatch[1], cpvMatch[2]
34	+	if name == "" \|\| version == "" {
35	+	log.WithFields("path", reader.Location.RealPath).Warnf("failed to parse portage name and version")
36	+	return nil, nil, nil
37	+	}
38	+
39	+	p := pkg.Package{
40	+	Name: name,
41	+	Version: version,
42	+	PURL: packageURL(name, version),
43	+	Locations: source.NewLocationSet(),
44	+	Type: pkg.PortagePkg,
45	+	MetadataType: pkg.PortageMetadataType,
46	+	Metadata: pkg.PortageMetadata{
47	+	// ensure the default value for a collection is never nil since this may be shown as JSON
48	+	Files: make([]pkg.PortageFileRecord, 0),
49	+	},
50	+	}
51	+	addLicenses(resolver, reader.Location, &p)
52	+	addSize(resolver, reader.Location, &p)
53	+	addFiles(resolver, reader.Location, &p)
54	+
55	+	p.SetID()
56	+
57	+	return []pkg.Package{p}, nil, nil
58	+	}
59	+
60	+	func addFiles(resolver source.FileResolver, dbLocation source.Location, p *pkg.Package) {
61	+	contentsReader, err := resolver.FileContentsByLocation(dbLocation)
62	+	if err != nil {
63	+	log.WithFields("path", dbLocation.RealPath).Warnf("failed to fetch portage contents (package=%s): %+v", p.Name, err)
64	+	return
65	+	}
66	+
67	+	entry, ok := p.Metadata.(pkg.PortageMetadata)
68	+	if !ok {
69	+	return
70	+	}
71	+
72	+	scanner := bufio.NewScanner(contentsReader)
73	+	for scanner.Scan() {
74	+	line := strings.Trim(scanner.Text(), "\n")
75	+	fields := strings.Split(line, " ")
76	+
77	+	if fields[0] == "obj" {
78	+	record := pkg.PortageFileRecord{
79	+	Path: fields[1],
80	+	}
81	+	record.Digest = &file.Digest{
82	+	Algorithm: "md5",
83	+	Value: fields[2],
84	+	}
85	+	entry.Files = append(entry.Files, record)
86	+	}
87	+	}
88	+
89	+	p.Metadata = entry
90	+	p.Locations.Add(dbLocation)
91	+	}
92	+
93	+	func addLicenses(resolver source.FileResolver, dbLocation source.Location, p *pkg.Package) {
94	+	parentPath := filepath.Dir(dbLocation.RealPath)
95	+
96	+	location := resolver.RelativeFileByPath(dbLocation, path.Join(parentPath, "LICENSE"))
97	+
98	+	if location == nil {
99	+	return
100	+	}
101	+
102	+	licenseReader, err := resolver.FileContentsByLocation(*location)
103	+	if err != nil {
104	+	log.WithFields("path", dbLocation.RealPath).Warnf("failed to fetch portage LICENSE: %+v", err)
105	+	return
106	+	}
107	+
108	+	findings := internal.NewStringSet()
109	+	scanner := bufio.NewScanner(licenseReader)
110	+	scanner.Split(bufio.ScanWords)
111	+	for scanner.Scan() {
112	+	token := scanner.Text()
113	+	if token != "\|\|" && token != "(" && token != ")" {
114	+	findings.Add(token)
115	+	}
116	+	}
117	+	licenses := findings.ToSlice()
118	+	sort.Strings(licenses)
119	+	p.Licenses = licenses
120	+	p.Locations.Add(*location)
121	+	}
122	+
123	+	func addSize(resolver source.FileResolver, dbLocation source.Location, p *pkg.Package) {
124	+	parentPath := filepath.Dir(dbLocation.RealPath)
125	+
126	+	location := resolver.RelativeFileByPath(dbLocation, path.Join(parentPath, "SIZE"))
127	+
128	+	if location == nil {
129	+	return
130	+	}
131	+
132	+	entry, ok := p.Metadata.(pkg.PortageMetadata)
133	+	if !ok {
134	+	return
135	+	}
136	+
137	+	sizeReader, err := resolver.FileContentsByLocation(*location)
138	+	if err != nil {
139	+	log.WithFields("name", p.Name).Warnf("failed to fetch portage SIZE: %+v", err)
140	+	return
141	+	}
142	+
143	+	scanner := bufio.NewScanner(sizeReader)
144	+	for scanner.Scan() {
145	+	line := strings.Trim(scanner.Text(), "\n")
146	+	size, err := strconv.Atoi(line)
147	+	if err == nil {
148	+	entry.InstalledSize = size
149	+	}
150	+	}
151	+
152	+	p.Metadata = entry
153	+	p.Locations.Add(*location)
154	+	}
155	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/portage/test-fixtures/glob-paths/var/db/pkg/x/y/CONTENTS

1

+

bogus contents

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/python/cataloger.go

		skipped 3 lines
4	4		"github.com/anchore/syft/syft/pkg/cataloger/generic"
5	5		)
6	6
7		-	const (
8		-	eggMetadataGlob = "*/egg-info/PKG-INFO"
9		-	eggFileMetadataGlob = "*/.egg-info"
10		-	wheelMetadataGlob = "*/dist-info/METADATA"
11		-	)
	7	+	const eggInfoGlob = "*/.egg-info"
12	8
13	9		// NewPythonIndexCataloger returns a new cataloger for python packages referenced from poetry lock files, requirements.txt files, and setup.py files.
14	10		func NewPythonIndexCataloger() *generic.Cataloger {
		skipped 7 lines
22	18		// NewPythonPackageCataloger returns a new cataloger for python packages within egg or wheel installation directories.
23	19		func NewPythonPackageCataloger() *generic.Cataloger {
24	20		return generic.NewCataloger("python-package-cataloger").
25		-	WithParserByGlobs(parseWheelOrEgg, eggMetadataGlob, eggFileMetadataGlob, wheelMetadataGlob)
	21	+	WithParserByGlobs(parseWheelOrEgg, eggInfoGlob, "*/dist-info/METADATA", "*/egg-info/PKG-INFO")
26	22		}
27	23

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/python/cataloger_test.go

		skipped 2 lines
3	3		import (
4	4		"testing"
5	5
	6	+	"github.com/stretchr/testify/require"
	7	+
6	8		"github.com/anchore/syft/syft/pkg"
7	9		"github.com/anchore/syft/syft/pkg/cataloger/internal/pkgtest"
8	10		"github.com/anchore/syft/syft/source"
		skipped 183 lines
192	194		resolver := source.NewMockResolverForPaths(test.fixtures...)
193	195
194	196		locations, err := resolver.FilesByPath(test.fixtures...)
195		-	if err != nil {
196		-	t.Fatal(err)
197		-	}
	197	+	require.NoError(t, err)
198	198
199	199		test.expectedPackage.Locations = source.NewLocationSet(locations...)
200	200
		skipped 22 lines
223	223		resolver := source.NewMockResolverForPaths(test.MetadataFixture)
224	224
225	225		actual, _, err := NewPythonPackageCataloger().Catalog(resolver)
226		-	if err != nil {
227		-	t.Fatalf("failed to catalog python package: %+v", err)
228		-	}
	226	+	require.NoError(t, err)
229	227
230	228		if len(actual) != 0 {
231	229		t.Fatalf("Expected 0 packages but found: %d", len(actual))
		skipped 2 lines
234	232		}
235	233		}
236	234
	235	+	func Test_IndexCataloger_Globs(t *testing.T) {
	236	+	tests := []struct {
	237	+	name string
	238	+	fixture string
	239	+	expected []string
	240	+	}{
	241	+	{
	242	+	name: "obtain index files",
	243	+	fixture: "test-fixtures/glob-paths",
	244	+	expected: []string{
	245	+	"src/requirements.txt",
	246	+	"src/extra-requirements.txt",
	247	+	"src/requirements-dev.txt",
	248	+	"src/1-requirements-dev.txt",
	249	+	"src/setup.py",
	250	+	"src/poetry.lock",
	251	+	"src/Pipfile.lock",
	252	+	},
	253	+	},
	254	+	}
	255	+
	256	+	for _, test := range tests {
	257	+	t.Run(test.name, func(t *testing.T) {
	258	+	pkgtest.NewCatalogTester().
	259	+	FromDirectory(t, test.fixture).
	260	+	ExpectsResolverContentQueries(test.expected).
	261	+	TestCataloger(t, NewPythonIndexCataloger())
	262	+	})
	263	+	}
	264	+	}
	265	+
	266	+	func Test_PackageCataloger_Globs(t *testing.T) {
	267	+	tests := []struct {
	268	+	name string
	269	+	fixture string
	270	+	expected []string
	271	+	}{
	272	+	{
	273	+	name: "obtain index files",
	274	+	fixture: "test-fixtures/glob-paths",
	275	+	expected: []string{
	276	+	"site-packages/x.dist-info/METADATA",
	277	+	"site-packages/y.egg-info/PKG-INFO",
	278	+	"site-packages/z.egg-info",
	279	+	},
	280	+	},
	281	+	}
	282	+
	283	+	for _, test := range tests {
	284	+	t.Run(test.name, func(t *testing.T) {
	285	+	pkgtest.NewCatalogTester().
	286	+	FromDirectory(t, test.fixture).
	287	+	ExpectsResolverContentQueries(test.expected).
	288	+	TestCataloger(t, NewPythonPackageCataloger())
	289	+	})
	290	+	}
	291	+	}
	292	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/python/parse_wheel_egg_metadata.go

		skipped 80 lines
81	81		// of egg metadata (as opposed to a directory that contains more metadata
82	82		// files).
83	83		func isEggRegularFile(path string) bool {
84		-	return file.GlobMatch(eggFileMetadataGlob, path)
	84	+	return file.GlobMatch(eggInfoGlob, path)
85	85		}
86	86
87	87		// determineSitePackagesRootPath returns the path of the site packages root,
		skipped 25 lines

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/python/test-fixtures/glob-paths/site-packages/x.dist-info/METADATA

	1	+	bogus
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/python/test-fixtures/glob-paths/site-packages/y.egg-info/PKG-INFO

	1	+	bogus
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/python/test-fixtures/glob-paths/site-packages/z.egg-info

1

+

bogus

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/python/test-fixtures/glob-paths/src/1-requirements-dev.txt

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/python/test-fixtures/glob-paths/src/Pipfile.lock

	1	+	bogus
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/python/test-fixtures/glob-paths/src/extra-requirements.txt

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/python/test-fixtures/glob-paths/src/poetry.lock

	1	+	bogus
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/python/test-fixtures/glob-paths/src/requirements-dev.txt

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/python/test-fixtures/glob-paths/src/requirements.txt

	1	+	example archive
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/python/test-fixtures/glob-paths/src/setup.py

	1	+	bogus
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/rpm/cataloger_test.go

1	+	package rpm
2	+
3	+	import (
4	+	"testing"
5	+
6	+	"github.com/anchore/syft/syft/pkg/cataloger/internal/pkgtest"
7	+	)
8	+
9	+	func Test_DBCataloger_Globs(t *testing.T) {
10	+	tests := []struct {
11	+	name string
12	+	fixture string
13	+	expected []string
14	+	}{
15	+	{
16	+	name: "obtain DB files",
17	+	fixture: "test-fixtures/glob-paths",
18	+	expected: []string{
19	+	"var/lib/rpm/Packages",
20	+	"var/lib/rpm/Packages.db",
21	+	"var/lib/rpm/rpmdb.sqlite",
22	+	"var/lib/rpmmanifest/container-manifest-2",
23	+	},
24	+	},
25	+	}
26	+
27	+	for _, test := range tests {
28	+	t.Run(test.name, func(t *testing.T) {
29	+	pkgtest.NewCatalogTester().
30	+	FromDirectory(t, test.fixture).
31	+	ExpectsResolverContentQueries(test.expected).
32	+	TestCataloger(t, NewRpmDBCataloger())
33	+	})
34	+	}
35	+	}
36	+
37	+	func Test_RPMFileCataloger_Globs(t *testing.T) {
38	+	tests := []struct {
39	+	name string
40	+	fixture string
41	+	expected []string
42	+	}{
43	+	{
44	+	name: "obtain rpm files",
45	+	fixture: "test-fixtures/glob-paths",
46	+	expected: []string{
47	+	"dive-0.10.0.rpm",
48	+	},
49	+	},
50	+	}
51	+
52	+	for _, test := range tests {
53	+	t.Run(test.name, func(t *testing.T) {
54	+	pkgtest.NewCatalogTester().
55	+	FromDirectory(t, test.fixture).
56	+	ExpectsResolverContentQueries(test.expected).
57	+	TestCataloger(t, NewFileCataloger())
58	+	})
59	+	}
60	+	}
61	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/rpm/parse_rpm_db_test.go

		skipped 18 lines
19	19		ignorePaths bool
20	20		}
21	21
	22	+	func (r rpmdbTestFileResolverMock) FilesByExtension(extensions ...string) ([]source.Location, error) {
	23	+	panic("not implemented")
	24	+	}
	25	+
	26	+	func (r rpmdbTestFileResolverMock) FilesByBasename(filenames ...string) ([]source.Location, error) {
	27	+	panic("not implemented")
	28	+	}
	29	+
	30	+	func (r rpmdbTestFileResolverMock) FilesByBasenameGlob(globs ...string) ([]source.Location, error) {
	31	+	panic("not implemented")
	32	+	}
	33	+
22	34		func (r rpmdbTestFileResolverMock) FileContentsByLocation(location source.Location) (io.ReadCloser, error) {
23		-	//TODO implement me
24		-	panic("implement me")
	35	+	panic("not implemented")
25	36		}
26	37
27	38		func (r rpmdbTestFileResolverMock) AllLocations() <-chan source.Location {
28		-	//TODO implement me
29		-	panic("implement me")
	39	+	panic("not implemented")
30	40		}
31	41
32	42		func (r rpmdbTestFileResolverMock) FileMetadataByLocation(location source.Location) (source.FileMetadata, error) {
33		-	//TODO implement me
34		-	panic("implement me")
	43	+	panic("not implemented")
35	44		}
36	45
37	46		func newTestFileResolver(ignorePaths bool) *rpmdbTestFileResolverMock {
		skipped 170 lines

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/rpm/test-fixtures/glob-paths/dive-0.10.0.rpm

1

+

bogus

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/rpm/test-fixtures/glob-paths/var/lib/rpm/Packages

1

+

bogus

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/rpm/test-fixtures/glob-paths/var/lib/rpm/Packages.db

1

+

bogus

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/rpm/test-fixtures/glob-paths/var/lib/rpm/rpmdb.sqlite

1

+

bogus

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/rpm/test-fixtures/glob-paths/var/lib/rpmmanifest/container-manifest-2

1

+

bogus

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/ruby/catalogers_test.go

1	+	package ruby
2	+
3	+	import (
4	+	"testing"
5	+
6	+	"github.com/anchore/syft/syft/pkg/cataloger/internal/pkgtest"
7	+	)
8	+
9	+	func Test_GemFileLock_Globs(t *testing.T) {
10	+	tests := []struct {
11	+	name string
12	+	fixture string
13	+	expected []string
14	+	}{
15	+	{
16	+	name: "obtain gemfile lock files",
17	+	fixture: "test-fixtures/glob-paths",
18	+	expected: []string{
19	+	"src/Gemfile.lock",
20	+	},
21	+	},
22	+	}
23	+
24	+	for _, test := range tests {
25	+	t.Run(test.name, func(t *testing.T) {
26	+	pkgtest.NewCatalogTester().
27	+	FromDirectory(t, test.fixture).
28	+	ExpectsResolverContentQueries(test.expected).
29	+	TestCataloger(t, NewGemFileLockCataloger())
30	+	})
31	+	}
32	+	}
33	+
34	+	func Test_GemSpec_Globs(t *testing.T) {
35	+	tests := []struct {
36	+	name string
37	+	fixture string
38	+	expected []string
39	+	}{
40	+	{
41	+	name: "obtain gemspec files",
42	+	fixture: "test-fixtures/glob-paths",
43	+	expected: []string{
44	+	"specifications/root.gemspec",
45	+	"specifications/pkg/nested.gemspec",
46	+	},
47	+	},
48	+	}
49	+
50	+	for _, test := range tests {
51	+	t.Run(test.name, func(t *testing.T) {
52	+	pkgtest.NewCatalogTester().
53	+	FromDirectory(t, test.fixture).
54	+	ExpectsResolverContentQueries(test.expected).
55	+	TestCataloger(t, NewGemSpecCataloger())
56	+	})
57	+	}
58	+	}
59	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/ruby/test-fixtures/glob-paths/specifications/pkg/nested.gemspec

1

+

bogus

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/ruby/test-fixtures/glob-paths/specifications/root.gemspec

1

+

bogus

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/ruby/test-fixtures/glob-paths/src/Gemfile.lock

1

+

bogus

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/rust/cataloger_test.go

		skipped 49 lines
50	50		TestCataloger(t, NewAuditBinaryCataloger())
51	51		}
52	52
	53	+	func Test_CargoLockCataloger_Globs(t *testing.T) {
	54	+	tests := []struct {
	55	+	name string
	56	+	fixture string
	57	+	expected []string
	58	+	}{
	59	+	{
	60	+	name: "obtain Cargo.lock files",
	61	+	fixture: "test-fixtures/glob-paths",
	62	+	expected: []string{
	63	+	"src/Cargo.lock",
	64	+	},
	65	+	},
	66	+	}
	67	+
	68	+	for _, test := range tests {
	69	+	t.Run(test.name, func(t *testing.T) {
	70	+	pkgtest.NewCatalogTester().
	71	+	FromDirectory(t, test.fixture).
	72	+	ExpectsResolverContentQueries(test.expected).
	73	+	TestCataloger(t, NewCargoLockCataloger())
	74	+	})
	75	+	}
	76	+	}
	77	+
	78	+	func Test_AuditBinaryCataloger_Globs(t *testing.T) {
	79	+	tests := []struct {
	80	+	name string
	81	+	fixture string
	82	+	expected []string
	83	+	}{
	84	+	{
	85	+	name: "obtain audit binary files",
	86	+	fixture: "test-fixtures/glob-paths",
	87	+	expected: []string{
	88	+	"partial-binary",
	89	+	},
	90	+	},
	91	+	}
	92	+
	93	+	for _, test := range tests {
	94	+	t.Run(test.name, func(t *testing.T) {
	95	+	pkgtest.NewCatalogTester().
	96	+	FromDirectory(t, test.fixture).
	97	+	ExpectsResolverContentQueries(test.expected).
	98	+	TestCataloger(t, NewAuditBinaryCataloger())
	99	+	})
	100	+	}
	101	+	}
	102	+

syft/pkg/cataloger/rust/test-fixtures/glob-paths/partial-binary

Binary file.

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/rust/test-fixtures/glob-paths/src/Cargo.lock

1

+

bogus

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/sbom/cataloger_test.go

		skipped 290 lines
291	291		}
292	292		}
293	293
	294	+	func Test_Cataloger_Globs(t *testing.T) {
	295	+	tests := []struct {
	296	+	name string
	297	+	fixture string
	298	+	expected []string
	299	+	}{
	300	+	{
	301	+	name: "obtain sbom files",
	302	+	fixture: "test-fixtures/glob-paths",
	303	+	expected: []string{
	304	+	"bom",
	305	+	"sbom",
	306	+	"app.syft.json",
	307	+	"app.bom",
	308	+	"app.sbom",
	309	+	"app.cdx",
	310	+	"app.spdx",
	311	+	"app.bom.json",
	312	+	"app.sbom.json",
	313	+	"app.cdx.json",
	314	+	"app.spdx.json",
	315	+	},
	316	+	},
	317	+	}
	318	+
	319	+	for _, test := range tests {
	320	+	t.Run(test.name, func(t *testing.T) {
	321	+	pkgtest.NewCatalogTester().
	322	+	FromDirectory(t, test.fixture).
	323	+	ExpectsResolverContentQueries(test.expected).
	324	+	TestCataloger(t, NewSBOMCataloger())
	325	+	})
	326	+	}
	327	+	}
	328	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/sbom/test-fixtures/glob-paths/app.bom

	1	+	bogus
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/sbom/test-fixtures/glob-paths/app.bom.json

	1	+	bogus
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/sbom/test-fixtures/glob-paths/app.cdx

	1	+	bogus
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/sbom/test-fixtures/glob-paths/app.cdx.json

	1	+	bogus
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/sbom/test-fixtures/glob-paths/app.sbom

	1	+	bogus
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/sbom/test-fixtures/glob-paths/app.sbom.json

	1	+	bogus
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/sbom/test-fixtures/glob-paths/app.spdx

	1	+	bogus
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/sbom/test-fixtures/glob-paths/app.spdx.json

	1	+	bogus
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/sbom/test-fixtures/glob-paths/app.syft.json

	1	+	bogus
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/sbom/test-fixtures/glob-paths/bom

	1	+	bogus
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/sbom/test-fixtures/glob-paths/sbom

	1	+	bogus
	2	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/swift/cataloger_test.go

1	+	package swift
2	+
3	+	import (
4	+	"testing"
5	+
6	+	"github.com/anchore/syft/syft/pkg/cataloger/internal/pkgtest"
7	+	)
8	+
9	+	func Test_Cataloger_Globs(t *testing.T) {
10	+	tests := []struct {
11	+	name string
12	+	fixture string
13	+	expected []string
14	+	}{
15	+	{
16	+	name: "obtain swift files",
17	+	fixture: "test-fixtures/glob-paths",
18	+	expected: []string{
19	+	"src/Podfile.lock",
20	+	},
21	+	},
22	+	}
23	+
24	+	for _, test := range tests {
25	+	t.Run(test.name, func(t *testing.T) {
26	+	pkgtest.NewCatalogTester().
27	+	FromDirectory(t, test.fixture).
28	+	ExpectsResolverContentQueries(test.expected).
29	+	TestCataloger(t, NewCocoapodsCataloger())
30	+	})
31	+	}
32	+	}
33	+

■ ■ ■ ■ ■ ■

syft/pkg/cataloger/swift/test-fixtures/glob-paths/src/Podfile.lock

1

+

bogus

■ ■ ■ ■ ■ ■

syft/source/all_layers_resolver.go

1	-	package source
2	-
3	-	import (
4	-	"archive/tar"
5	-	"fmt"
6	-	"io"
7	-
8	-	"github.com/anchore/stereoscope/pkg/file"
9	-	"github.com/anchore/stereoscope/pkg/filetree"
10	-	"github.com/anchore/stereoscope/pkg/image"
11	-	"github.com/anchore/syft/internal/log"
12	-	)
13	-
14	-	var _ FileResolver = (*allLayersResolver)(nil)
15	-
16	-	// allLayersResolver implements path and content access for the AllLayers source option for container image data sources.
17	-	type allLayersResolver struct {
18	-	img *image.Image
19	-	layers []int
20	-	}
21	-
22	-	// newAllLayersResolver returns a new resolver from the perspective of all image layers for the given image.
23	-	func newAllLayersResolver(img image.Image) (allLayersResolver, error) {
24	-	if len(img.Layers) == 0 {
25	-	return nil, fmt.Errorf("the image does not contain any layers")
26	-	}
27	-
28	-	var layers = make([]int, 0)
29	-	for idx := range img.Layers {
30	-	layers = append(layers, idx)
31	-	}
32	-	return &allLayersResolver{
33	-	img: img,
34	-	layers: layers,
35	-	}, nil
36	-	}
37	-
38	-	// HasPath indicates if the given path exists in the underlying source.
39	-	func (r *allLayersResolver) HasPath(path string) bool {
40	-	p := file.Path(path)
41	-	for _, layerIdx := range r.layers {
42	-	tree := r.img.Layers[layerIdx].Tree
43	-	if tree.HasPath(p) {
44	-	return true
45	-	}
46	-	}
47	-	return false
48	-	}
49	-
50	-	func (r *allLayersResolver) fileByRef(ref file.Reference, uniqueFileIDs file.ReferenceSet, layerIdx int) ([]file.Reference, error) {
51	-	uniqueFiles := make([]file.Reference, 0)
52	-
53	-	// since there is potentially considerable work for each symlink/hardlink that needs to be resolved, let's check to see if this is a symlink/hardlink first
54	-	entry, err := r.img.FileCatalog.Get(ref)
55	-	if err != nil {
56	-	return nil, fmt.Errorf("unable to fetch metadata (ref=%+v): %w", ref, err)
57	-	}
58	-
59	-	if entry.Metadata.TypeFlag == tar.TypeLink \|\| entry.Metadata.TypeFlag == tar.TypeSymlink {
60	-	// a link may resolve in this layer or higher, assuming a squashed tree is used to search
61	-	// we should search all possible resolutions within the valid source
62	-	for _, subLayerIdx := range r.layers[layerIdx:] {
63	-	resolvedRef, err := r.img.ResolveLinkByLayerSquash(ref, subLayerIdx)
64	-	if err != nil {
65	-	return nil, fmt.Errorf("failed to resolve link from layer (layer=%d ref=%+v): %w", subLayerIdx, ref, err)
66	-	}
67	-	if resolvedRef != nil && !uniqueFileIDs.Contains(*resolvedRef) {
68	-	uniqueFileIDs.Add(*resolvedRef)
69	-	uniqueFiles = append(uniqueFiles, *resolvedRef)
70	-	}
71	-	}
72	-	} else if !uniqueFileIDs.Contains(ref) {
73	-	uniqueFileIDs.Add(ref)
74	-	uniqueFiles = append(uniqueFiles, ref)
75	-	}
76	-
77	-	return uniqueFiles, nil
78	-	}
79	-
80	-	// FilesByPath returns all file.References that match the given paths from any layer in the image.
81	-	func (r *allLayersResolver) FilesByPath(paths ...string) ([]Location, error) {
82	-	uniqueFileIDs := file.NewFileReferenceSet()
83	-	uniqueLocations := make([]Location, 0)
84	-
85	-	for _, path := range paths {
86	-	for idx, layerIdx := range r.layers {
87	-	tree := r.img.Layers[layerIdx].Tree
88	-	_, ref, err := tree.File(file.Path(path), filetree.FollowBasenameLinks, filetree.DoNotFollowDeadBasenameLinks)
89	-	if err != nil {
90	-	return nil, err
91	-	}
92	-	if ref == nil {
93	-	// no file found, keep looking through layers
94	-	continue
95	-	}
96	-
97	-	// don't consider directories (special case: there is no path information for /)
98	-	if ref.RealPath == "/" {
99	-	continue
100	-	} else if r.img.FileCatalog.Exists(*ref) {
101	-	metadata, err := r.img.FileCatalog.Get(*ref)
102	-	if err != nil {
103	-	return nil, fmt.Errorf("unable to get file metadata for path=%q: %w", ref.RealPath, err)
104	-	}
105	-	if metadata.Metadata.IsDir {
106	-	continue
107	-	}
108	-	}
109	-
110	-	results, err := r.fileByRef(*ref, uniqueFileIDs, idx)
111	-	if err != nil {
112	-	return nil, err
113	-	}
114	-	for _, result := range results {
115	-	uniqueLocations = append(uniqueLocations, NewLocationFromImage(path, result, r.img))
116	-	}
117	-	}
118	-	}
119	-	return uniqueLocations, nil
120	-	}
121	-
122	-	// FilesByGlob returns all file.References that match the given path glob pattern from any layer in the image.
123	-	func (r *allLayersResolver) FilesByGlob(patterns ...string) ([]Location, error) {
124	-	uniqueFileIDs := file.NewFileReferenceSet()
125	-	uniqueLocations := make([]Location, 0)
126	-
127	-	for _, pattern := range patterns {
128	-	for idx, layerIdx := range r.layers {
129	-	results, err := r.img.Layers[layerIdx].SquashedTree.FilesByGlob(pattern, filetree.FollowBasenameLinks, filetree.DoNotFollowDeadBasenameLinks)
130	-	if err != nil {
131	-	return nil, fmt.Errorf("failed to resolve files by glob (%s): %w", pattern, err)
132	-	}
133	-
134	-	for _, result := range results {
135	-	// don't consider directories (special case: there is no path information for /)
136	-	if result.RealPath == "/" {
137	-	continue
138	-	} else if r.img.FileCatalog.Exists(result.Reference) {
139	-	metadata, err := r.img.FileCatalog.Get(result.Reference)
140	-	if err != nil {
141	-	return nil, fmt.Errorf("unable to get file metadata for path=%q: %w", result.MatchPath, err)
142	-	}
143	-	if metadata.Metadata.IsDir {
144	-	continue
145	-	}
146	-	}
147	-
148	-	refResults, err := r.fileByRef(result.Reference, uniqueFileIDs, idx)
149	-	if err != nil {
150	-	return nil, err
151	-	}
152	-	for _, refResult := range refResults {
153	-	uniqueLocations = append(uniqueLocations, NewLocationFromImage(string(result.MatchPath), refResult, r.img))
154	-	}
155	-	}
156	-	}
157	-	}
158	-
159	-	return uniqueLocations, nil
160	-	}
161	-
162	-	// RelativeFileByPath fetches a single file at the given path relative to the layer squash of the given reference.
163	-	// This is helpful when attempting to find a file that is in the same layer or lower as another file.
164	-	func (r allLayersResolver) RelativeFileByPath(location Location, path string) Location {
165	-	entry, err := r.img.FileCatalog.Get(location.ref)
166	-	if err != nil {
167	-	return nil
168	-	}
169	-
170	-	exists, relativeRef, err := entry.Layer.SquashedTree.File(file.Path(path), filetree.FollowBasenameLinks)
171	-	if err != nil {
172	-	log.Errorf("failed to find path=%q in squash: %+w", path, err)
173	-	return nil
174	-	}
175	-	if !exists && relativeRef == nil {
176	-	return nil
177	-	}
178	-
179	-	relativeLocation := NewLocationFromImage(path, *relativeRef, r.img)
180	-
181	-	return &relativeLocation
182	-	}
183	-
184	-	// FileContentsByLocation fetches file contents for a single file reference, irregardless of the source layer.
185	-	// If the path does not exist an error is returned.
186	-	func (r *allLayersResolver) FileContentsByLocation(location Location) (io.ReadCloser, error) {
187	-	entry, err := r.img.FileCatalog.Get(location.ref)
188	-	if err != nil {
189	-	return nil, fmt.Errorf("unable to get metadata for path=%q from file catalog: %w", location.RealPath, err)
190	-	}
191	-
192	-	switch entry.Metadata.TypeFlag {
193	-	case tar.TypeSymlink, tar.TypeLink:
194	-	// the location we are searching may be a symlink, we should always work with the resolved file
195	-	newLocation := r.RelativeFileByPath(location, location.VirtualPath)
196	-	if newLocation == nil {
197	-	// this is a dead link
198	-	return nil, fmt.Errorf("no contents for location=%q", location.VirtualPath)
199	-	}
200	-	location = *newLocation
201	-	}
202	-
203	-	return r.img.FileContentsByRef(location.ref)
204	-	}
205	-
206	-	func (r *allLayersResolver) FilesByMIMEType(types ...string) ([]Location, error) {
207	-	var locations []Location
208	-	for _, layerIdx := range r.layers {
209	-	layer := r.img.Layers[layerIdx]
210	-
211	-	refs, err := layer.FilesByMIMEType(types...)
212	-	if err != nil {
213	-	return nil, err
214	-	}
215	-
216	-	for _, ref := range refs {
217	-	locations = append(locations, NewLocationFromImage(string(ref.RealPath), ref, r.img))
218	-	}
219	-	}
220	-
221	-	return locations, nil
222	-	}
223	-
224	-	func (r *allLayersResolver) AllLocations() <-chan Location {
225	-	results := make(chan Location)
226	-	go func() {
227	-	defer close(results)
228	-	for _, layerIdx := range r.layers {
229	-	tree := r.img.Layers[layerIdx].Tree
230	-	for _, ref := range tree.AllFiles(file.AllTypes...) {
231	-	results <- NewLocationFromImage(string(ref.RealPath), ref, r.img)
232	-	}
233	-	}
234	-	}()
235	-	return results
236	-	}
237	-
238	-	func (r *allLayersResolver) FileMetadataByLocation(location Location) (FileMetadata, error) {
239	-	return fileMetadataByLocation(r.img, location)
240	-	}
241	-

■ ■ ■ ■ ■ ■

syft/source/directory_indexer.go

1	+	package source
2	+
3	+	import (
4	+	"errors"
5	+	"fmt"
6	+	"io/fs"
7	+	"os"
8	+	"path"
9	+	"path/filepath"
10	+	"runtime"
11	+
12	+	"github.com/wagoodman/go-partybus"
13	+	"github.com/wagoodman/go-progress"
14	+
15	+	"github.com/anchore/stereoscope/pkg/file"
16	+	"github.com/anchore/stereoscope/pkg/filetree"
17	+	"github.com/anchore/syft/internal"
18	+	"github.com/anchore/syft/internal/bus"
19	+	"github.com/anchore/syft/internal/log"
20	+	"github.com/anchore/syft/syft/event"
21	+	)
22	+
23	+	type pathIndexVisitor func(string, os.FileInfo, error) error
24	+
25	+	type directoryIndexer struct {
26	+	path string
27	+	base string
28	+	pathIndexVisitors []pathIndexVisitor
29	+	errPaths map[string]error
30	+	tree filetree.ReadWriter
31	+	index filetree.Index
32	+	}
33	+
34	+	func newDirectoryIndexer(path, base string, visitors ...pathIndexVisitor) *directoryIndexer {
35	+	i := &directoryIndexer{
36	+	path: path,
37	+	base: base,
38	+	tree: filetree.New(),
39	+	index: filetree.NewIndex(),
40	+	pathIndexVisitors: append([]pathIndexVisitor{requireFileInfo, disallowByFileType, disallowUnixSystemRuntimePath}, visitors...),
41	+	errPaths: make(map[string]error),
42	+	}
43	+
44	+	// these additional stateful visitors should be the first thing considered when walking / indexing
45	+	i.pathIndexVisitors = append(
46	+	[]pathIndexVisitor{
47	+	i.disallowRevisitingVisitor,
48	+	i.disallowFileAccessErr,
49	+	},
50	+	i.pathIndexVisitors...,
51	+	)
52	+
53	+	return i
54	+	}
55	+
56	+	func (r *directoryIndexer) build() (filetree.Reader, filetree.IndexReader, error) {
57	+	return r.tree, r.index, indexAllRoots(r.path, r.indexTree)
58	+	}
59	+
60	+	func indexAllRoots(root string, indexer func(string, *progress.Stage) ([]string, error)) error {
61	+	// why account for multiple roots? To cover cases when there is a symlink that references above the root path,
62	+	// in which case we need to additionally index where the link resolves to. it's for this reason why the filetree
63	+	// must be relative to the root of the filesystem (and not just relative to the given path).
64	+	pathsToIndex := []string{root}
65	+	fullPathsMap := map[string]struct{}{}
66	+
67	+	stager, prog := indexingProgress(root)
68	+	defer prog.SetCompleted()
69	+	loop:
70	+	for {
71	+	var currentPath string
72	+	switch len(pathsToIndex) {
73	+	case 0:
74	+	break loop
75	+	case 1:
76	+	currentPath, pathsToIndex = pathsToIndex[0], nil
77	+	default:
78	+	currentPath, pathsToIndex = pathsToIndex[0], pathsToIndex[1:]
79	+	}
80	+
81	+	additionalRoots, err := indexer(currentPath, stager)
82	+	if err != nil {
83	+	return fmt.Errorf("unable to index filesystem path=%q: %w", currentPath, err)
84	+	}
85	+
86	+	for _, newRoot := range additionalRoots {
87	+	if _, ok := fullPathsMap[newRoot]; !ok {
88	+	fullPathsMap[newRoot] = struct{}{}
89	+	pathsToIndex = append(pathsToIndex, newRoot)
90	+	}
91	+	}
92	+	}
93	+
94	+	return nil
95	+	}
96	+
97	+	func (r directoryIndexer) indexTree(root string, stager progress.Stage) ([]string, error) {
98	+	log.WithFields("path", root).Trace("indexing filetree")
99	+
100	+	var roots []string
101	+	var err error
102	+
103	+	root, err = filepath.Abs(root)
104	+	if err != nil {
105	+	return nil, err
106	+	}
107	+
108	+	// we want to be able to index single files with the directory resolver. However, we should also allow for attempting
109	+	// to index paths that do not exist (that is, a root that does not exist is not an error case that should stop indexing).
110	+	// For this reason we look for an opportunity to discover if the given root is a file, and if so add a single root,
111	+	// but continue forth with index regardless if the given root path exists or not.
112	+	fi, err := os.Stat(root)
113	+	if err != nil && fi != nil && !fi.IsDir() {
114	+	// note: we want to index the path regardless of an error stat-ing the path
115	+	newRoot, _ := r.indexPath(root, fi, nil)
116	+	if newRoot != "" {
117	+	roots = append(roots, newRoot)
118	+	}
119	+	return roots, nil
120	+	}
121	+
122	+	err = filepath.Walk(root,
123	+	func(path string, info os.FileInfo, err error) error {
124	+	stager.Current = path
125	+
126	+	newRoot, err := r.indexPath(path, info, err)
127	+
128	+	if err != nil {
129	+	return err
130	+	}
131	+
132	+	if newRoot != "" {
133	+	roots = append(roots, newRoot)
134	+	}
135	+
136	+	return nil
137	+	})
138	+
139	+	if err != nil {
140	+	return nil, fmt.Errorf("unable to index root=%q: %w", root, err)
141	+	}
142	+
143	+	return roots, nil
144	+	}
145	+
146	+	func (r *directoryIndexer) indexPath(path string, info os.FileInfo, err error) (string, error) {
147	+	// ignore any path which a filter function returns true
148	+	for _, filterFn := range r.pathIndexVisitors {
149	+	if filterFn == nil {
150	+	continue
151	+	}
152	+
153	+	if filterErr := filterFn(path, info, err); filterErr != nil {
154	+	if errors.Is(filterErr, fs.SkipDir) {
155	+	// signal to walk() to skip this directory entirely (even if we're processing a file)
156	+	return "", filterErr
157	+	}
158	+	// skip this path but don't affect walk() trajectory
159	+	return "", nil
160	+	}
161	+	}
162	+
163	+	if info == nil {
164	+	// walk may not be able to provide a FileInfo object, don't allow for this to stop indexing; keep track of the paths and continue.
165	+	r.errPaths[path] = fmt.Errorf("no file info observable at path=%q", path)
166	+	return "", nil
167	+	}
168	+
169	+	// here we check to see if we need to normalize paths to posix on the way in coming from windows
170	+	if runtime.GOOS == WindowsOS {
171	+	path = windowsToPosix(path)
172	+	}
173	+
174	+	newRoot, err := r.addPathToIndex(path, info)
175	+	if r.isFileAccessErr(path, err) {
176	+	return "", nil
177	+	}
178	+
179	+	return newRoot, nil
180	+	}
181	+
182	+	func (r *directoryIndexer) disallowFileAccessErr(path string, _ os.FileInfo, err error) error {
183	+	if r.isFileAccessErr(path, err) {
184	+	return errSkipPath
185	+	}
186	+	return nil
187	+	}
188	+
189	+	func (r *directoryIndexer) isFileAccessErr(path string, err error) bool {
190	+	// don't allow for errors to stop indexing, keep track of the paths and continue.
191	+	if err != nil {
192	+	log.Warnf("unable to access path=%q: %+v", path, err)
193	+	r.errPaths[path] = err
194	+	return true
195	+	}
196	+	return false
197	+	}
198	+
199	+	func (r directoryIndexer) addPathToIndex(p string, info os.FileInfo) (string, error) {
200	+	switch t := file.TypeFromMode(info.Mode()); t {
201	+	case file.TypeSymLink:
202	+	return r.addSymlinkToIndex(p, info)
203	+	case file.TypeDirectory:
204	+	return "", r.addDirectoryToIndex(p, info)
205	+	case file.TypeRegular:
206	+	return "", r.addFileToIndex(p, info)
207	+	default:
208	+	return "", fmt.Errorf("unsupported file type: %s", t)
209	+	}
210	+	}
211	+
212	+	func (r directoryIndexer) addDirectoryToIndex(p string, info os.FileInfo) error {
213	+	ref, err := r.tree.AddDir(file.Path(p))
214	+	if err != nil {
215	+	return err
216	+	}
217	+
218	+	metadata := file.NewMetadataFromPath(p, info)
219	+	r.index.Add(*ref, metadata)
220	+
221	+	return nil
222	+	}
223	+
224	+	func (r directoryIndexer) addFileToIndex(p string, info os.FileInfo) error {
225	+	ref, err := r.tree.AddFile(file.Path(p))
226	+	if err != nil {
227	+	return err
228	+	}
229	+
230	+	metadata := file.NewMetadataFromPath(p, info)
231	+	r.index.Add(*ref, metadata)
232	+
233	+	return nil
234	+	}
235	+
236	+	func (r directoryIndexer) addSymlinkToIndex(p string, info os.FileInfo) (string, error) {
237	+	linkTarget, err := os.Readlink(p)
238	+	if err != nil {
239	+	return "", fmt.Errorf("unable to readlink for path=%q: %w", p, err)
240	+	}
241	+
242	+	if filepath.IsAbs(linkTarget) {
243	+	// if the link is absolute (e.g, /bin/ls -> /bin/busybox) we need to
244	+	// resolve relative to the root of the base directory
245	+	linkTarget = filepath.Join(r.base, filepath.Clean(linkTarget))
246	+	} else {
247	+	// if the link is not absolute (e.g, /dev/stderr -> fd/2 ) we need to
248	+	// resolve it relative to the directory in question (e.g. resolve to
249	+	// /dev/fd/2)
250	+	if r.base == "" {
251	+	linkTarget = filepath.Join(filepath.Dir(p), linkTarget)
252	+	} else {
253	+	// if the base is set, then we first need to resolve the link,
254	+	// before finding it's location in the base
255	+	dir, err := filepath.Rel(r.base, filepath.Dir(p))
256	+	if err != nil {
257	+	return "", fmt.Errorf("unable to resolve relative path for path=%q: %w", p, err)
258	+	}
259	+	linkTarget = filepath.Join(r.base, filepath.Clean(filepath.Join("/", dir, linkTarget)))
260	+	}
261	+	}
262	+
263	+	ref, err := r.tree.AddSymLink(file.Path(p), file.Path(linkTarget))
264	+	if err != nil {
265	+	return "", err
266	+	}
267	+
268	+	targetAbsPath := linkTarget
269	+	if !filepath.IsAbs(targetAbsPath) {
270	+	targetAbsPath = filepath.Clean(filepath.Join(path.Dir(p), linkTarget))
271	+	}
272	+
273	+	metadata := file.NewMetadataFromPath(p, info)
274	+	metadata.LinkDestination = linkTarget
275	+	r.index.Add(*ref, metadata)
276	+
277	+	return targetAbsPath, nil
278	+	}
279	+
280	+	func (r directoryIndexer) hasBeenIndexed(p string) (bool, *file.Metadata) {
281	+	filePath := file.Path(p)
282	+	if !r.tree.HasPath(filePath) {
283	+	return false, nil
284	+	}
285	+
286	+	exists, ref, err := r.tree.File(filePath)
287	+	if err != nil \|\| !exists \|\| !ref.HasReference() {
288	+	return false, nil
289	+	}
290	+
291	+	// cases like "/" will be in the tree, but not been indexed yet (a special case). We want to capture
292	+	// these cases as new paths to index.
293	+	if !ref.HasReference() {
294	+	return false, nil
295	+	}
296	+
297	+	entry, err := r.index.Get(*ref.Reference)
298	+	if err != nil {
299	+	return false, nil
300	+	}
301	+
302	+	return true, &entry.Metadata
303	+	}
304	+
305	+	func (r *directoryIndexer) disallowRevisitingVisitor(path string, _ os.FileInfo, _ error) error {
306	+	// this prevents visiting:
307	+	// - link destinations twice, once for the real file and another through the virtual path
308	+	// - infinite link cycles
309	+	if indexed, metadata := r.hasBeenIndexed(path); indexed {
310	+	if metadata.IsDir {
311	+	// signal to walk() that we should skip this directory entirely
312	+	return fs.SkipDir
313	+	}
314	+	return errSkipPath
315	+	}
316	+	return nil
317	+	}
318	+
319	+	func disallowUnixSystemRuntimePath(path string, _ os.FileInfo, _ error) error {
320	+	if internal.HasAnyOfPrefixes(path, unixSystemRuntimePrefixes...) {
321	+	return fs.SkipDir
322	+	}
323	+	return nil
324	+	}
325	+
326	+	func disallowByFileType(_ string, info os.FileInfo, _ error) error {
327	+	if info == nil {
328	+	// we can't filter out by filetype for non-existent files
329	+	return nil
330	+	}
331	+	switch file.TypeFromMode(info.Mode()) {
332	+	case file.TypeCharacterDevice, file.TypeSocket, file.TypeBlockDevice, file.TypeFIFO, file.TypeIrregular:
333	+	return errSkipPath
334	+	// note: symlinks that point to these files may still get by.
335	+	// We handle this later in processing to help prevent against infinite links traversal.
336	+	}
337	+
338	+	return nil
339	+	}
340	+
341	+	func requireFileInfo(_ string, info os.FileInfo, _ error) error {
342	+	if info == nil {
343	+	return errSkipPath
344	+	}
345	+	return nil
346	+	}
347	+
348	+	func indexingProgress(path string) (progress.Stage, progress.Manual) {
349	+	stage := &progress.Stage{}
350	+	prog := &progress.Manual{
351	+	Total: -1,
352	+	}
353	+
354	+	bus.Publish(partybus.Event{
355	+	Type: event.FileIndexingStarted,
356	+	Source: path,
357	+	Value: struct {
358	+	progress.Stager
359	+	progress.Progressable
360	+	}{
361	+	Stager: progress.Stager(stage),
362	+	Progressable: prog,
363	+	},
364	+	})
365	+
366	+	return stage, prog
367	+	}
368	+

■ ■ ■ ■ ■ ■

syft/source/directory_indexer_test.go

1	+	package source
2	+
3	+	import (
4	+	"io/fs"
5	+	"os"
6	+	"path"
7	+	"sort"
8	+	"strings"
9	+	"testing"
10	+
11	+	"github.com/google/go-cmp/cmp"
12	+	"github.com/scylladb/go-set/strset"
13	+	"github.com/stretchr/testify/assert"
14	+	"github.com/stretchr/testify/require"
15	+	"github.com/wagoodman/go-progress"
16	+
17	+	"github.com/anchore/stereoscope/pkg/file"
18	+	)
19	+
20	+	type indexerMock struct {
21	+	observedRoots []string
22	+	additionalRoots map[string][]string
23	+	}
24	+
25	+	func (m indexerMock) indexer(s string, _ progress.Stage) ([]string, error) {
26	+	m.observedRoots = append(m.observedRoots, s)
27	+	return m.additionalRoots[s], nil
28	+	}
29	+
30	+	func Test_indexAllRoots(t *testing.T) {
31	+	tests := []struct {
32	+	name string
33	+	root string
34	+	mock indexerMock
35	+	expectedRoots []string
36	+	}{
37	+	{
38	+	name: "no additional roots",
39	+	root: "a/place",
40	+	mock: indexerMock{
41	+	additionalRoots: make(map[string][]string),
42	+	},
43	+	expectedRoots: []string{
44	+	"a/place",
45	+	},
46	+	},
47	+	{
48	+	name: "additional roots from a single call",
49	+	root: "a/place",
50	+	mock: indexerMock{
51	+	additionalRoots: map[string][]string{
52	+	"a/place": {
53	+	"another/place",
54	+	"yet-another/place",
55	+	},
56	+	},
57	+	},
58	+	expectedRoots: []string{
59	+	"a/place",
60	+	"another/place",
61	+	"yet-another/place",
62	+	},
63	+	},
64	+	{
65	+	name: "additional roots from a multiple calls",
66	+	root: "a/place",
67	+	mock: indexerMock{
68	+	additionalRoots: map[string][]string{
69	+	"a/place": {
70	+	"another/place",
71	+	"yet-another/place",
72	+	},
73	+	"yet-another/place": {
74	+	"a-quiet-place-2",
75	+	"a-final/place",
76	+	},
77	+	},
78	+	},
79	+	expectedRoots: []string{
80	+	"a/place",
81	+	"another/place",
82	+	"yet-another/place",
83	+	"a-quiet-place-2",
84	+	"a-final/place",
85	+	},
86	+	},
87	+	}
88	+
89	+	for _, test := range tests {
90	+	t.Run(test.name, func(t *testing.T) {
91	+	assert.NoError(t, indexAllRoots(test.root, test.mock.indexer))
92	+	})
93	+	}
94	+	}
95	+
96	+	func TestDirectoryIndexer_handleFileAccessErr(t *testing.T) {
97	+	tests := []struct {
98	+	name string
99	+	input error
100	+	expectedPathTracked bool
101	+	}{
102	+	{
103	+	name: "permission error does not propagate",
104	+	input: os.ErrPermission,
105	+	expectedPathTracked: true,
106	+	},
107	+	{
108	+	name: "file does not exist error does not propagate",
109	+	input: os.ErrNotExist,
110	+	expectedPathTracked: true,
111	+	},
112	+	{
113	+	name: "non-permission errors are tracked",
114	+	input: os.ErrInvalid,
115	+	expectedPathTracked: true,
116	+	},
117	+	{
118	+	name: "non-errors ignored",
119	+	input: nil,
120	+	expectedPathTracked: false,
121	+	},
122	+	}
123	+
124	+	for _, test := range tests {
125	+	t.Run(test.name, func(t *testing.T) {
126	+	r := directoryIndexer{
127	+	errPaths: make(map[string]error),
128	+	}
129	+	p := "a/place"
130	+	assert.Equal(t, r.isFileAccessErr(p, test.input), test.expectedPathTracked)
131	+	_, exists := r.errPaths[p]
132	+	assert.Equal(t, test.expectedPathTracked, exists)
133	+	})
134	+	}
135	+	}
136	+
137	+	func TestDirectoryIndexer_IncludeRootPathInIndex(t *testing.T) {
138	+	filterFn := func(path string, _ os.FileInfo, _ error) error {
139	+	if path != "/" {
140	+	return fs.SkipDir
141	+	}
142	+	return nil
143	+	}
144	+
145	+	indexer := newDirectoryIndexer("/", "", filterFn)
146	+	tree, index, err := indexer.build()
147	+	require.NoError(t, err)
148	+
149	+	exists, ref, err := tree.File(file.Path("/"))
150	+	require.NoError(t, err)
151	+	require.NotNil(t, ref)
152	+	assert.True(t, exists)
153	+
154	+	_, err = index.Get(*ref.Reference)
155	+	require.NoError(t, err)
156	+	}
157	+
158	+	func TestDirectoryIndexer_indexPath_skipsNilFileInfo(t *testing.T) {
159	+	// TODO: Ideally we can use an OS abstraction, which would obviate the need for real FS setup.
160	+	tempFile, err := os.CreateTemp("", "")
161	+	require.NoError(t, err)
162	+
163	+	indexer := newDirectoryIndexer(tempFile.Name(), "")
164	+
165	+	t.Run("filtering path with nil os.FileInfo", func(t *testing.T) {
166	+	assert.NotPanics(t, func() {
167	+	_, err := indexer.indexPath("/dont-care", nil, nil)
168	+	assert.NoError(t, err)
169	+	assert.False(t, indexer.tree.HasPath("/dont-care"))
170	+	})
171	+	})
172	+	}
173	+
174	+	func TestDirectoryIndexer_index(t *testing.T) {
175	+	// note: this test is testing the effects from newDirectoryResolver, indexTree, and addPathToIndex
176	+	indexer := newDirectoryIndexer("test-fixtures/system_paths/target", "")
177	+	tree, index, err := indexer.build()
178	+	require.NoError(t, err)
179	+
180	+	tests := []struct {
181	+	name string
182	+	path string
183	+	}{
184	+	{
185	+	name: "has dir",
186	+	path: "test-fixtures/system_paths/target/home",
187	+	},
188	+	{
189	+	name: "has path",
190	+	path: "test-fixtures/system_paths/target/home/place",
191	+	},
192	+	{
193	+	name: "has symlink",
194	+	path: "test-fixtures/system_paths/target/link/a-symlink",
195	+	},
196	+	{
197	+	name: "has symlink target",
198	+	path: "test-fixtures/system_paths/outside_root/link_target/place",
199	+	},
200	+	}
201	+	for _, test := range tests {
202	+	t.Run(test.name, func(t *testing.T) {
203	+	info, err := os.Stat(test.path)
204	+	assert.NoError(t, err)
205	+
206	+	// note: the index uses absolute paths, so assertions MUST keep this in mind
207	+	cwd, err := os.Getwd()
208	+	require.NoError(t, err)
209	+
210	+	p := file.Path(path.Join(cwd, test.path))
211	+	assert.Equal(t, true, tree.HasPath(p))
212	+	exists, ref, err := tree.File(p)
213	+	assert.Equal(t, true, exists)
214	+	if assert.NoError(t, err) {
215	+	return
216	+	}
217	+
218	+	entry, err := index.Get(*ref.Reference)
219	+	require.NoError(t, err)
220	+	assert.Equal(t, info.Mode(), entry.Mode)
221	+	})
222	+	}
223	+	}
224	+
225	+	func TestDirectoryIndexer_SkipsAlreadyVisitedLinkDestinations(t *testing.T) {
226	+	var observedPaths []string
227	+	pathObserver := func(p string, _ os.FileInfo, _ error) error {
228	+	fields := strings.Split(p, "test-fixtures/symlinks-prune-indexing")
229	+	if len(fields) != 2 {
230	+	t.Fatalf("unable to parse path: %s", p)
231	+	}
232	+	clean := strings.TrimLeft(fields[1], "/")
233	+	if clean != "" {
234	+	observedPaths = append(observedPaths, clean)
235	+	}
236	+	return nil
237	+	}
238	+	resolver := newDirectoryIndexer("./test-fixtures/symlinks-prune-indexing", "")
239	+	// we want to cut ahead of any possible filters to see what paths are considered for indexing (closest to walking)
240	+	resolver.pathIndexVisitors = append([]pathIndexVisitor{pathObserver}, resolver.pathIndexVisitors...)
241	+
242	+	// note: this test is NOT about the effects left on the tree or the index, but rather the WHICH paths that are
243	+	// considered for indexing and HOW traversal prunes paths that have already been visited
244	+	_, _, err := resolver.build()
245	+	require.NoError(t, err)
246	+
247	+	expected := []string{
248	+	"before-path",
249	+	"c-file.txt",
250	+	"c-path",
251	+	"path",
252	+	"path/1",
253	+	"path/1/2",
254	+	"path/1/2/3",
255	+	"path/1/2/3/4",
256	+	"path/1/2/3/4/dont-index-me-twice.txt",
257	+	"path/5",
258	+	"path/5/6",
259	+	"path/5/6/7",
260	+	"path/5/6/7/8",
261	+	"path/5/6/7/8/dont-index-me-twice-either.txt",
262	+	"path/file.txt",
263	+	// everything below is after the original tree is indexed, and we are now indexing additional roots from symlinks
264	+	"path", // considered from symlink before-path, but pruned
265	+	"before-path/file.txt", // considered from symlink c-file.txt, but pruned
266	+	"before-path", // considered from symlink c-path, but pruned
267	+	}
268	+
269	+	assert.Equal(t, expected, observedPaths, "visited paths differ \n %s", cmp.Diff(expected, observedPaths))
270	+
271	+	}
272	+
273	+	func TestDirectoryIndexer_IndexesAllTypes(t *testing.T) {
274	+	indexer := newDirectoryIndexer("./test-fixtures/symlinks-prune-indexing", "")
275	+
276	+	tree, index, err := indexer.build()
277	+	require.NoError(t, err)
278	+
279	+	allRefs := tree.AllFiles(file.AllTypes()...)
280	+	var pathRefs []file.Reference
281	+	paths := strset.New()
282	+	for _, ref := range allRefs {
283	+	fields := strings.Split(string(ref.RealPath), "test-fixtures/symlinks-prune-indexing")
284	+	if len(fields) != 2 {
285	+	t.Fatalf("unable to parse path: %s", ref.RealPath)
286	+	}
287	+	clean := strings.TrimLeft(fields[1], "/")
288	+	if clean == "" {
289	+	continue
290	+	}
291	+	paths.Add(clean)
292	+	pathRefs = append(pathRefs, ref)
293	+	}
294	+
295	+	pathsList := paths.List()
296	+	sort.Strings(pathsList)
297	+
298	+	expected := []string{
299	+	"before-path", // link
300	+	"c-file.txt", // link
301	+	"c-path", // link
302	+	"path", // dir
303	+	"path/1", // dir
304	+	"path/1/2", // dir
305	+	"path/1/2/3", // dir
306	+	"path/1/2/3/4", // dir
307	+	"path/1/2/3/4/dont-index-me-twice.txt", // file
308	+	"path/5", // dir
309	+	"path/5/6", // dir
310	+	"path/5/6/7", // dir
311	+	"path/5/6/7/8", // dir
312	+	"path/5/6/7/8/dont-index-me-twice-either.txt", // file
313	+	"path/file.txt", // file
314	+	}
315	+	expectedSet := strset.New(expected...)
316	+
317	+	// make certain all expected paths are in the tree (and no extra ones are their either)
318	+
319	+	assert.True(t, paths.IsEqual(expectedSet), "expected all paths to be indexed, but found different paths: \n%s", cmp.Diff(expected, pathsList))
320	+
321	+	// make certain that the paths are also in the file index
322	+
323	+	for _, ref := range pathRefs {
324	+	_, err := index.Get(ref)
325	+	require.NoError(t, err)
326	+	}
327	+
328	+	}
329	+

■ ■ ■ ■ ■ ■

syft/source/directory_resolver.go

		skipped 3 lines
4	4		"errors"
5	5		"fmt"
6	6		"io"
7		-	"io/fs"
8	7		"os"
9	8		"path"
10	9		"path/filepath"
11	10		"runtime"
12	11		"strings"
13		-
14		-	"github.com/wagoodman/go-partybus"
15		-	"github.com/wagoodman/go-progress"
16	12
17	13		"github.com/anchore/stereoscope/pkg/file"
18	14		"github.com/anchore/stereoscope/pkg/filetree"
19		-	"github.com/anchore/syft/internal"
20		-	"github.com/anchore/syft/internal/bus"
21	15		"github.com/anchore/syft/internal/log"
22		-	"github.com/anchore/syft/syft/event"
23	16		)
24	17
25	18		const WindowsOS = "windows"
		skipped 4 lines
30	23		"/sys",
31	24		}
32	25
33		-	var _ FileResolver = (*directoryResolver)(nil)
	26	+	var errSkipPath = errors.New("skip path")
34	27
35		-	type pathFilterFn func(string, os.FileInfo) bool
	28	+	var _ FileResolver = (*directoryResolver)(nil)
36	29
37	30		// directoryResolver implements path and content access for the directory data source.
38	31		type directoryResolver struct {
		skipped 1 lines
40	33		base string
41	34		currentWdRelativeToRoot string
42	35		currentWd string
43		-	fileTree *filetree.FileTree
44		-	metadata map[file.ID]FileMetadata
45		-	// TODO: wire up to report these paths in the json report
46		-	pathFilterFns []pathFilterFn
47		-	refsByMIMEType map[string][]file.Reference
48		-	errPaths map[string]error
	36	+	tree filetree.Reader
	37	+	index filetree.IndexReader
	38	+	searchContext filetree.Searcher
	39	+	indexer *directoryIndexer
49	40		}
50	41
51		-	func newDirectoryResolver(root string, base string, pathFilters ...pathFilterFn) (*directoryResolver, error) {
	42	+	func newDirectoryResolver(root string, base string, pathFilters ...pathIndexVisitor) (*directoryResolver, error) {
	43	+	r, err := newDirectoryResolverWithoutIndex(root, base, pathFilters...)
	44	+	if err != nil {
	45	+	return nil, err
	46	+	}
	47	+
	48	+	return r, r.buildIndex()
	49	+	}
	50	+
	51	+	func newDirectoryResolverWithoutIndex(root string, base string, pathFilters ...pathIndexVisitor) (*directoryResolver, error) {
52	52		currentWD, err := os.Getwd()
53	53		if err != nil {
54	54		return nil, fmt.Errorf("could not get CWD: %w", err)
		skipped 32 lines
87	87		currentWdRelRoot = filepath.Clean(cleanRoot)
88	88		}
89	89
90		-	resolver := directoryResolver{
	90	+	return &directoryResolver{
91	91		path: cleanRoot,
92	92		base: cleanBase,
93	93		currentWd: cleanCWD,
94	94		currentWdRelativeToRoot: currentWdRelRoot,
95		-	fileTree: filetree.NewFileTree(),
96		-	metadata: make(map[file.ID]FileMetadata),
97		-	pathFilterFns: append([]pathFilterFn{isUnallowableFileType, isUnixSystemRuntimePath}, pathFilters...),
98		-	refsByMIMEType: make(map[string][]file.Reference),
99		-	errPaths: make(map[string]error),
100		-	}
101		-
102		-	return &resolver, indexAllRoots(cleanRoot, resolver.indexTree)
	95	+	tree: filetree.New(),
	96	+	index: filetree.NewIndex(),
	97	+	indexer: newDirectoryIndexer(cleanRoot, cleanBase, pathFilters...),
	98	+	}, nil
103	99		}
104	100
105		-	func (r directoryResolver) indexTree(root string, stager progress.Stage) ([]string, error) {
106		-	log.Debugf("indexing filesystem path=%q", root)
107		-
108		-	var roots []string
109		-	var err error
110		-
111		-	root, err = filepath.Abs(root)
112		-	if err != nil {
113		-	return nil, err
	101	+	func (r *directoryResolver) buildIndex() error {
	102	+	if r.indexer == nil {
	103	+	return fmt.Errorf("no directory indexer configured")
114	104		}
115		-
116		-	// we want to be able to index single files with the directory resolver. However, we should also allow for attempting
117		-	// to index paths that do not exist (that is, a root that does not exist is not an error case that should stop indexing).
118		-	// For this reason we look for an opportunity to discover if the given root is a file, and if so add a single root,
119		-	// but continue forth with index regardless if the given root path exists or not.
120		-	fi, err := os.Stat(root)
121		-	if err != nil && fi != nil && !fi.IsDir() {
122		-	// note: we want to index the path regardless of an error stat-ing the path
123		-	newRoot, _ := r.indexPath(root, fi, nil)
124		-	if newRoot != "" {
125		-	roots = append(roots, newRoot)
126		-	}
127		-	return roots, nil
128		-	}
129		-
130		-	return roots, filepath.Walk(root,
131		-	func(path string, info os.FileInfo, err error) error {
132		-	stager.Current = path
133		-
134		-	newRoot, err := r.indexPath(path, info, err)
135		-
136		-	if err != nil {
137		-	return err
138		-	}
139		-
140		-	if newRoot != "" {
141		-	roots = append(roots, newRoot)
142		-	}
143		-
144		-	return nil
145		-	})
146		-	}
147		-
148		-	func (r *directoryResolver) indexPath(path string, info os.FileInfo, err error) (string, error) {
149		-	// link cycles could cause a revisit --we should not allow this
150		-	if r.hasBeenIndexed(path) {
151		-	return "", nil
152		-	}
153		-
154		-	// ignore any path which a filter function returns true
155		-	for _, filterFn := range r.pathFilterFns {
156		-	if filterFn != nil && filterFn(path, info) {
157		-	if info != nil && info.IsDir() {
158		-	return "", fs.SkipDir
159		-	}
160		-	return "", nil
161		-	}
162		-	}
163		-
164		-	if r.isFileAccessErr(path, err) {
165		-	return "", nil
166		-	}
167		-
168		-	if info == nil {
169		-	// walk may not be able to provide a FileInfo object, don't allow for this to stop indexing; keep track of the paths and continue.
170		-	r.errPaths[path] = fmt.Errorf("no file info observable at path=%q", path)
171		-	return "", nil
172		-	}
173		-
174		-	// here we check to see if we need to normalize paths to posix on the way in coming from windows
175		-	if runtime.GOOS == WindowsOS {
176		-	path = windowsToPosix(path)
177		-	}
178		-
179		-	newRoot, err := r.addPathToIndex(path, info)
180		-	if r.isFileAccessErr(path, err) {
181		-	return "", nil
182		-	}
183		-
184		-	return newRoot, nil
185		-	}
186		-
187		-	func (r *directoryResolver) isFileAccessErr(path string, err error) bool {
188		-	// don't allow for errors to stop indexing, keep track of the paths and continue.
189		-	if err != nil {
190		-	log.Warnf("unable to access path=%q: %+v", path, err)
191		-	r.errPaths[path] = err
192		-	return true
193		-	}
194		-	return false
195		-	}
196		-
197		-	func (r directoryResolver) addPathToIndex(p string, info os.FileInfo) (string, error) {
198		-	switch t := newFileTypeFromMode(info.Mode()); t {
199		-	case SymbolicLink:
200		-	return r.addSymlinkToIndex(p, info)
201		-	case Directory:
202		-	return "", r.addDirectoryToIndex(p, info)
203		-	case RegularFile:
204		-	return "", r.addFileToIndex(p, info)
205		-	default:
206		-	return "", fmt.Errorf("unsupported file type: %s", t)
207		-	}
208		-	}
209		-
210		-	func (r directoryResolver) hasBeenIndexed(p string) bool {
211		-	filePath := file.Path(p)
212		-	if !r.fileTree.HasPath(filePath) {
213		-	return false
214		-	}
215		-
216		-	exists, ref, err := r.fileTree.File(filePath)
217		-	if err != nil \|\| !exists \|\| ref == nil {
218		-	return false
219		-	}
220		-
221		-	// cases like "/" will be in the tree, but not been indexed yet (a special case). We want to capture
222		-	// these cases as new paths to index.
223		-	_, exists = r.metadata[ref.ID()]
224		-	return exists
225		-	}
226		-
227		-	func (r directoryResolver) addDirectoryToIndex(p string, info os.FileInfo) error {
228		-	ref, err := r.fileTree.AddDir(file.Path(p))
	105	+	tree, index, err := r.indexer.build()
229	106		if err != nil {
230	107		return err
231	108		}
232	109
233		-	location := NewLocationFromDirectory(p, *ref)
234		-	metadata := fileMetadataFromPath(p, info, r.isInIndex(location))
235		-	r.addFileMetadataToIndex(ref, metadata)
	110	+	r.tree = tree
	111	+	r.index = index
	112	+	r.searchContext = filetree.NewSearchContext(tree, index)
236	113
237	114		return nil
238	115		}
239	116
240		-	func (r directoryResolver) addFileToIndex(p string, info os.FileInfo) error {
241		-	ref, err := r.fileTree.AddFile(file.Path(p))
242		-	if err != nil {
243		-	return err
244		-	}
245		-
246		-	location := NewLocationFromDirectory(p, *ref)
247		-	metadata := fileMetadataFromPath(p, info, r.isInIndex(location))
248		-	r.addFileMetadataToIndex(ref, metadata)
249		-
250		-	return nil
251		-	}
252		-
253		-	func (r directoryResolver) addSymlinkToIndex(p string, info os.FileInfo) (string, error) {
254		-	var usedInfo = info
255		-
256		-	linkTarget, err := os.Readlink(p)
257		-	if err != nil {
258		-	return "", fmt.Errorf("unable to readlink for path=%q: %w", p, err)
259		-	}
260		-
261		-	if filepath.IsAbs(linkTarget) {
262		-	// if the link is absolute (e.g, /bin/ls -> /bin/busybox) we need to
263		-	// resolve relative to the root of the base directory
264		-	linkTarget = filepath.Join(r.base, filepath.Clean(linkTarget))
265		-	} else {
266		-	// if the link is not absolute (e.g, /dev/stderr -> fd/2 ) we need to
267		-	// resolve it relative to the directory in question (e.g. resolve to
268		-	// /dev/fd/2)
269		-	if r.base == "" {
270		-	linkTarget = filepath.Join(filepath.Dir(p), linkTarget)
271		-	} else {
272		-	// if the base is set, then we first need to resolve the link,
273		-	// before finding it's location in the base
274		-	dir, err := filepath.Rel(r.base, filepath.Dir(p))
275		-	if err != nil {
276		-	return "", fmt.Errorf("unable to resolve relative path for path=%q: %w", p, err)
277		-	}
278		-	linkTarget = filepath.Join(r.base, filepath.Clean(filepath.Join("/", dir, linkTarget)))
279		-	}
280		-	}
281		-
282		-	ref, err := r.fileTree.AddSymLink(file.Path(p), file.Path(linkTarget))
283		-	if err != nil {
284		-	return "", err
285		-	}
286		-
287		-	targetAbsPath := linkTarget
288		-	if !filepath.IsAbs(targetAbsPath) {
289		-	targetAbsPath = filepath.Clean(filepath.Join(path.Dir(p), linkTarget))
290		-	}
291		-
292		-	location := NewLocationFromDirectory(p, *ref)
293		-	location.VirtualPath = p
294		-	metadata := fileMetadataFromPath(p, usedInfo, r.isInIndex(location))
295		-	metadata.LinkDestination = linkTarget
296		-	r.addFileMetadataToIndex(ref, metadata)
297		-
298		-	return targetAbsPath, nil
299		-	}
300		-
301		-	func (r directoryResolver) addFileMetadataToIndex(ref *file.Reference, metadata FileMetadata) {
302		-	if ref != nil {
303		-	if metadata.MIMEType != "" {
304		-	r.refsByMIMEType[metadata.MIMEType] = append(r.refsByMIMEType[metadata.MIMEType], *ref)
305		-	}
306		-	r.metadata[ref.ID()] = metadata
307		-	}
308		-	}
309		-
310	117		func (r directoryResolver) requestPath(userPath string) (string, error) {
311	118		if filepath.IsAbs(userPath) {
312	119		// don't allow input to potentially hop above root path
		skipped 39 lines
352	159		if err != nil {
353	160		return false
354	161		}
355		-	return r.fileTree.HasPath(file.Path(requestPath))
	162	+	return r.tree.HasPath(file.Path(requestPath))
356	163		}
357	164
358	165		// Stringer to represent a directory path data source
		skipped 13 lines
372	179		}
373	180
374	181		// we should be resolving symlinks and preserving this information as a VirtualPath to the real file
375		-	exists, ref, err := r.fileTree.File(file.Path(userStrPath), filetree.FollowBasenameLinks)
	182	+	ref, err := r.searchContext.SearchByPath(userStrPath, filetree.FollowBasenameLinks)
376	183		if err != nil {
377	184		log.Tracef("unable to evaluate symlink for path=%q : %+v", userPath, err)
378	185		continue
379	186		}
380		-	if !exists {
	187	+
	188	+	if !ref.HasReference() {
381	189		continue
382	190		}
383	191
384		-	// TODO: why not use stored metadata?
385		-	fileMeta, err := os.Stat(string(ref.RealPath))
386		-	if errors.Is(err, os.ErrNotExist) {
387		-	// note: there are other kinds of errors other than os.ErrNotExist that may be given that is platform
388		-	// specific, but essentially hints at the same overall problem (that the path does not exist). Such an
389		-	// error could be syscall.ENOTDIR (see https://github.com/golang/go/issues/18974).
390		-	continue
391		-	} else if err != nil {
392		-	// we don't want to consider any other syscalls that may hint at non-existence of the file/dir as
393		-	// invalid paths. This logging statement is meant to raise IO or permissions related problems.
394		-	var pathErr *os.PathError
395		-	if !errors.As(err, &pathErr) {
396		-	log.Warnf("path is not valid (%s): %+v", ref.RealPath, err)
397		-	}
	192	+	entry, err := r.index.Get(*ref.Reference)
	193	+	if err != nil {
	194	+	log.Warnf("unable to get file by path=%q : %+v", userPath, err)
398	195		continue
399	196		}
400	197
401	198		// don't consider directories
402		-	if fileMeta.IsDir() {
	199	+	if entry.Metadata.IsDir {
403	200		continue
404	201		}
405	202
		skipped 1 lines
407	204		userStrPath = windowsToPosix(userStrPath)
408	205		}
409	206
410		-	loc := NewVirtualLocationFromDirectory(
411		-	r.responsePath(string(ref.RealPath)), // the actual path relative to the resolver root
412		-	r.responsePath(userStrPath), // the path used to access this file, relative to the resolver root
413		-	*ref,
414		-	)
415		-	references = append(references, loc)
	207	+	if ref.HasReference() {
	208	+	references = append(references,
	209	+	NewVirtualLocationFromDirectory(
	210	+	r.responsePath(string(ref.RealPath)), // the actual path relative to the resolver root
	211	+	r.responsePath(userStrPath), // the path used to access this file, relative to the resolver root
	212	+	*ref.Reference,
	213	+	),
	214	+	)
	215	+	}
416	216		}
417	217
418	218		return references, nil
		skipped 1 lines
420	220
421	221		// FilesByGlob returns all file.References that match the given path glob pattern from any layer in the image.
422	222		func (r directoryResolver) FilesByGlob(patterns ...string) ([]Location, error) {
423		-	result := make([]Location, 0)
	223	+	uniqueFileIDs := file.NewFileReferenceSet()
	224	+	uniqueLocations := make([]Location, 0)
424	225
425	226		for _, pattern := range patterns {
426		-	globResults, err := r.fileTree.FilesByGlob(pattern, filetree.FollowBasenameLinks)
	227	+	refVias, err := r.searchContext.SearchByGlob(pattern, filetree.FollowBasenameLinks)
427	228		if err != nil {
428	229		return nil, err
429	230		}
430		-	for _, globResult := range globResults {
	231	+	for _, refVia := range refVias {
	232	+	if !refVia.HasReference() \|\| uniqueFileIDs.Contains(*refVia.Reference) {
	233	+	continue
	234	+	}
	235	+	entry, err := r.index.Get(*refVia.Reference)
	236	+	if err != nil {
	237	+	return nil, fmt.Errorf("unable to get file metadata for reference %s: %w", refVia.Reference.RealPath, err)
	238	+	}
	239	+
	240	+	// don't consider directories
	241	+	if entry.Metadata.IsDir {
	242	+	continue
	243	+	}
	244	+
431	245		loc := NewVirtualLocationFromDirectory(
432		-	r.responsePath(string(globResult.Reference.RealPath)), // the actual path relative to the resolver root
433		-	r.responsePath(string(globResult.MatchPath)), // the path used to access this file, relative to the resolver root
434		-	globResult.Reference,
	246	+	r.responsePath(string(refVia.Reference.RealPath)), // the actual path relative to the resolver root
	247	+	r.responsePath(string(refVia.RequestPath)), // the path used to access this file, relative to the resolver root
	248	+	*refVia.Reference,
435	249		)
436		-	result = append(result, loc)
	250	+	uniqueFileIDs.Add(*refVia.Reference)
	251	+	uniqueLocations = append(uniqueLocations, loc)
437	252		}
438	253		}
439	254
440		-	return result, nil
	255	+	return uniqueLocations, nil
441	256		}
442	257
443	258		// RelativeFileByPath fetches a single file at the given path relative to the layer squash of the given reference.
		skipped 17 lines
461	276		if location.ref.RealPath == "" {
462	277		return nil, errors.New("empty path given")
463	278		}
464		-	if !r.isInIndex(location) {
465		-	// this is in cases where paths have been explicitly excluded from the tree index. In which case
466		-	// we should DENY all content requests. Why? These paths have been indicated to be inaccessible (either
467		-	// by preference or these files are not readable by the current user).
468		-	return nil, fmt.Errorf("file content is inaccessible path=%q", location.ref.RealPath)
	279	+
	280	+	entry, err := r.index.Get(location.ref)
	281	+	if err != nil {
	282	+	return nil, err
469	283		}
	284	+
	285	+	// don't consider directories
	286	+	if entry.Type == file.TypeDirectory {
	287	+	return nil, fmt.Errorf("cannot read contents of non-file %q", location.ref.RealPath)
	288	+	}
	289	+
470	290		// RealPath is posix so for windows directory resolver we need to translate
471	291		// to its true on disk path.
472	292		filePath := string(location.ref.RealPath)
473	293		if runtime.GOOS == WindowsOS {
474	294		filePath = posixToWindows(filePath)
475	295		}
476		-	return file.NewLazyReadCloser(filePath), nil
477		-	}
478	296
479		-	func (r directoryResolver) isInIndex(location Location) bool {
480		-	if location.ref.RealPath == "" {
481		-	return false
482		-	}
483		-	return r.fileTree.HasPath(location.ref.RealPath, filetree.FollowBasenameLinks)
	297	+	return file.NewLazyReadCloser(filePath), nil
484	298		}
485	299
486	300		func (r *directoryResolver) AllLocations() <-chan Location {
487	301		results := make(chan Location)
488	302		go func() {
489	303		defer close(results)
490		-	// this should be all non-directory types
491		-	for _, ref := range r.fileTree.AllFiles(file.TypeReg, file.TypeSymlink, file.TypeHardLink, file.TypeBlockDevice, file.TypeCharacterDevice, file.TypeFifo) {
	304	+	for _, ref := range r.tree.AllFiles(file.AllTypes()...) {
492	305		results <- NewLocationFromDirectory(r.responsePath(string(ref.RealPath)), ref)
493	306		}
494	307		}()
		skipped 1 lines
496	309		}
497	310
498	311		func (r *directoryResolver) FileMetadataByLocation(location Location) (FileMetadata, error) {
499		-	metadata, exists := r.metadata[location.ref.ID()]
500		-	if !exists {
	312	+	entry, err := r.index.Get(location.ref)
	313	+	if err != nil {
501	314		return FileMetadata{}, fmt.Errorf("location: %+v : %w", location, os.ErrNotExist)
502	315		}
503	316
504		-	return metadata, nil
	317	+	return entry.Metadata, nil
505	318		}
506	319
507	320		func (r *directoryResolver) FilesByMIMEType(types ...string) ([]Location, error) {
508		-	var locations []Location
509		-	for _, ty := range types {
510		-	if refs, ok := r.refsByMIMEType[ty]; ok {
511		-	for _, ref := range refs {
512		-	locations = append(locations, NewLocationFromDirectory(r.responsePath(string(ref.RealPath)), ref))
513		-	}
	321	+	uniqueFileIDs := file.NewFileReferenceSet()
	322	+	uniqueLocations := make([]Location, 0)
	323	+
	324	+	refVias, err := r.searchContext.SearchByMIMEType(types...)
	325	+	if err != nil {
	326	+	return nil, err
	327	+	}
	328	+	for _, refVia := range refVias {
	329	+	if !refVia.HasReference() {
	330	+	continue
514	331		}
	332	+	if uniqueFileIDs.Contains(*refVia.Reference) {
	333	+	continue
	334	+	}
	335	+	location := NewLocationFromDirectory(
	336	+	r.responsePath(string(refVia.Reference.RealPath)),
	337	+	*refVia.Reference,
	338	+	)
	339	+	uniqueFileIDs.Add(*refVia.Reference)
	340	+	uniqueLocations = append(uniqueLocations, location)
515	341		}
516		-	return locations, nil
	342	+
	343	+	return uniqueLocations, nil
517	344		}
518	345
519	346		func windowsToPosix(windowsPath string) (posixPath string) {
		skipped 23 lines
543	370		return filepath.Clean(volumeName + remainingTranslatedPath)
544	371		}
545	372
546		-	func isUnixSystemRuntimePath(path string, _ os.FileInfo) bool {
547		-	return internal.HasAnyOfPrefixes(path, unixSystemRuntimePrefixes...)
548		-	}
549		-
550		-	func isUnallowableFileType(_ string, info os.FileInfo) bool {
551		-	if info == nil {
552		-	// we can't filter out by filetype for non-existent files
553		-	return false
554		-	}
555		-	switch newFileTypeFromMode(info.Mode()) {
556		-	case CharacterDevice, Socket, BlockDevice, FIFONode, IrregularFile:
557		-	return true
558		-	// note: symlinks that point to these files may still get by.
559		-	// We handle this later in processing to help prevent against infinite links traversal.
560		-	}
561		-
562		-	return false
563		-	}
564		-
565		-	func indexAllRoots(root string, indexer func(string, *progress.Stage) ([]string, error)) error {
566		-	// why account for multiple roots? To cover cases when there is a symlink that references above the root path,
567		-	// in which case we need to additionally index where the link resolves to. it's for this reason why the filetree
568		-	// must be relative to the root of the filesystem (and not just relative to the given path).
569		-	pathsToIndex := []string{root}
570		-	fullPathsMap := map[string]struct{}{}
571		-
572		-	stager, prog := indexingProgress(root)
573		-	defer prog.SetCompleted()
574		-	loop:
575		-	for {
576		-	var currentPath string
577		-	switch len(pathsToIndex) {
578		-	case 0:
579		-	break loop
580		-	case 1:
581		-	currentPath, pathsToIndex = pathsToIndex[0], nil
582		-	default:
583		-	currentPath, pathsToIndex = pathsToIndex[0], pathsToIndex[1:]
584		-	}
585		-
586		-	additionalRoots, err := indexer(currentPath, stager)
587		-	if err != nil {
588		-	return fmt.Errorf("unable to index filesystem path=%q: %w", currentPath, err)
589		-	}
590		-
591		-	for _, newRoot := range additionalRoots {
592		-	if _, ok := fullPathsMap[newRoot]; !ok {
593		-	fullPathsMap[newRoot] = struct{}{}
594		-	pathsToIndex = append(pathsToIndex, newRoot)
595		-	}
596		-	}
597		-	}
598		-
599		-	return nil
600		-	}
601		-
602		-	func indexingProgress(path string) (progress.Stage, progress.Manual) {
603		-	stage := &progress.Stage{}
604		-	prog := &progress.Manual{
605		-	Total: -1,
606		-	}
607		-
608		-	bus.Publish(partybus.Event{
609		-	Type: event.FileIndexingStarted,
610		-	Source: path,
611		-	Value: struct {
612		-	progress.Stager
613		-	progress.Progressable
614		-	}{
615		-	Stager: progress.Stager(stage),
616		-	Progressable: prog,
617		-	},
618		-	})
619		-
620		-	return stage, prog
621		-	}
622		-

■ ■ ■ ■ ■ ■

syft/source/directory_resolver_test.go

		skipped 7 lines
8	8		"io/fs"
9	9		"io/ioutil"
10	10		"os"
11		-	"path"
12	11		"path/filepath"
13		-	"reflect"
	12	+	"sort"
14	13		"strings"
15	14		"testing"
16	15		"time"
17	16
	17	+	"github.com/google/go-cmp/cmp"
18	18		"github.com/scylladb/go-set/strset"
19	19		"github.com/stretchr/testify/assert"
20	20		"github.com/stretchr/testify/require"
21		-	"github.com/wagoodman/go-progress"
22	21
23	22		"github.com/anchore/stereoscope/pkg/file"
24	23		)
		skipped 277 lines
302	301		// let's make certain that "dev/place" is not ignored, since it is not "/dev/place"
303	302		resolver, err := newDirectoryResolver("test-fixtures/system_paths/target", "")
304	303		assert.NoError(t, err)
305		-	// ensure the correct filter function is wired up by default
306		-	expectedFn := reflect.ValueOf(isUnallowableFileType)
307		-	actualFn := reflect.ValueOf(resolver.pathFilterFns[0])
308		-	assert.Equal(t, expectedFn.Pointer(), actualFn.Pointer())
309	304
310	305		// all paths should be found (non filtering matches a path)
311	306		locations, err := resolver.FilesByGlob("**/place")
312	307		assert.NoError(t, err)
313	308		// 4: within target/
314		-	// 1: target/link --> relative path to "place"
	309	+	// 1: target/link --> relative path to "place" // NOTE: this is filtered out since it not unique relative to outside_root/link_target/place
315	310		// 1: outside_root/link_target/place
316		-	assert.Len(t, locations, 6)
	311	+	assert.Len(t, locations, 5)
317	312
318	313		// ensure that symlink indexing outside of root worked
319	314		testLocation := "test-fixtures/system_paths/outside_root/link_target/place"
		skipped 43 lines
363	358		tests := []struct {
364	359		name string
365	360		info os.FileInfo
366		-	expected bool
	361	+	expected error
367	362		}{
368	363		{
369	364		name: "regular file",
370	365		info: testFileInfo{
371	366		mode: 0,
372	367		},
373		-	expected: false,
374	368		},
375	369		{
376	370		name: "dir",
377	371		info: testFileInfo{
378	372		mode: os.ModeDir,
379	373		},
380		-	expected: false,
381	374		},
382	375		{
383	376		name: "symlink",
384	377		info: testFileInfo{
385	378		mode: os.ModeSymlink,
386	379		},
387		-	expected: false,
388	380		},
389	381		{
390	382		name: "socket",
391	383		info: testFileInfo{
392	384		mode: os.ModeSocket,
393	385		},
394		-	expected: true,
	386	+	expected: errSkipPath,
395	387		},
396	388		{
397	389		name: "named pipe",
398	390		info: testFileInfo{
399	391		mode: os.ModeNamedPipe,
400	392		},
401		-	expected: true,
	393	+	expected: errSkipPath,
402	394		},
403	395		{
404	396		name: "char device",
405	397		info: testFileInfo{
406	398		mode: os.ModeCharDevice,
407	399		},
408		-	expected: true,
	400	+	expected: errSkipPath,
409	401		},
410	402		{
411	403		name: "block device",
412	404		info: testFileInfo{
413	405		mode: os.ModeDevice,
414	406		},
415		-	expected: true,
	407	+	expected: errSkipPath,
416	408		},
417	409		{
418	410		name: "irregular",
419	411		info: testFileInfo{
420	412		mode: os.ModeIrregular,
421	413		},
422		-	expected: true,
423		-	},
424		-	}
425		-	for _, test := range tests {
426		-	t.Run(test.name, func(t *testing.T) {
427		-	assert.Equal(t, test.expected, isUnallowableFileType("dont/care", test.info))
428		-	})
429		-	}
430		-	}
431		-
432		-	func Test_directoryResolver_index(t *testing.T) {
433		-	// note: this test is testing the effects from newDirectoryResolver, indexTree, and addPathToIndex
434		-	r, err := newDirectoryResolver("test-fixtures/system_paths/target", "")
435		-	if err != nil {
436		-	t.Fatalf("unable to get indexed dir resolver: %+v", err)
437		-	}
438		-	tests := []struct {
439		-	name string
440		-	path string
441		-	}{
442		-	{
443		-	name: "has dir",
444		-	path: "test-fixtures/system_paths/target/home",
445		-	},
446		-	{
447		-	name: "has path",
448		-	path: "test-fixtures/system_paths/target/home/place",
449		-	},
450		-	{
451		-	name: "has symlink",
452		-	path: "test-fixtures/system_paths/target/link/a-symlink",
453		-	},
454		-	{
455		-	name: "has symlink target",
456		-	path: "test-fixtures/system_paths/outside_root/link_target/place",
457		-	},
458		-	}
459		-	for _, test := range tests {
460		-	t.Run(test.name, func(t *testing.T) {
461		-	info, err := os.Stat(test.path)
462		-	assert.NoError(t, err)
463		-
464		-	// note: the index uses absolute paths, so assertions MUST keep this in mind
465		-	cwd, err := os.Getwd()
466		-	require.NoError(t, err)
467		-
468		-	p := file.Path(path.Join(cwd, test.path))
469		-	assert.Equal(t, true, r.fileTree.HasPath(p))
470		-	exists, ref, err := r.fileTree.File(p)
471		-	assert.Equal(t, true, exists)
472		-	if assert.NoError(t, err) {
473		-	return
474		-	}
475		-	assert.Equal(t, info, r.metadata[ref.ID()])
476		-	})
477		-	}
478		-	}
479		-
480		-	func Test_handleFileAccessErr(t *testing.T) {
481		-	tests := []struct {
482		-	name string
483		-	input error
484		-	expectedPathTracked bool
485		-	}{
486		-	{
487		-	name: "permission error does not propagate",
488		-	input: os.ErrPermission,
489		-	expectedPathTracked: true,
490		-	},
491		-	{
492		-	name: "file does not exist error does not propagate",
493		-	input: os.ErrNotExist,
494		-	expectedPathTracked: true,
495		-	},
496		-	{
497		-	name: "non-permission errors are tracked",
498		-	input: os.ErrInvalid,
499		-	expectedPathTracked: true,
500		-	},
501		-	{
502		-	name: "non-errors ignored",
503		-	input: nil,
504		-	expectedPathTracked: false,
505		-	},
506		-	}
507		-
508		-	for _, test := range tests {
509		-	t.Run(test.name, func(t *testing.T) {
510		-	r := directoryResolver{
511		-	errPaths: make(map[string]error),
512		-	}
513		-	p := "a/place"
514		-	assert.Equal(t, r.isFileAccessErr(p, test.input), test.expectedPathTracked)
515		-	_, exists := r.errPaths[p]
516		-	assert.Equal(t, test.expectedPathTracked, exists)
517		-	})
518		-	}
519		-	}
520		-
521		-	type indexerMock struct {
522		-	observedRoots []string
523		-	additionalRoots map[string][]string
524		-	}
525		-
526		-	func (m indexerMock) indexer(s string, _ progress.Stage) ([]string, error) {
527		-	m.observedRoots = append(m.observedRoots, s)
528		-	return m.additionalRoots[s], nil
529		-	}
530		-
531		-	func Test_indexAllRoots(t *testing.T) {
532		-	tests := []struct {
533		-	name string
534		-	root string
535		-	mock indexerMock
536		-	expectedRoots []string
537		-	}{
538		-	{
539		-	name: "no additional roots",
540		-	root: "a/place",
541		-	mock: indexerMock{
542		-	additionalRoots: make(map[string][]string),
543		-	},
544		-	expectedRoots: []string{
545		-	"a/place",
546		-	},
547		-	},
548		-	{
549		-	name: "additional roots from a single call",
550		-	root: "a/place",
551		-	mock: indexerMock{
552		-	additionalRoots: map[string][]string{
553		-	"a/place": {
554		-	"another/place",
555		-	"yet-another/place",
556		-	},
557		-	},
558		-	},
559		-	expectedRoots: []string{
560		-	"a/place",
561		-	"another/place",
562		-	"yet-another/place",
563		-	},
564		-	},
565		-	{
566		-	name: "additional roots from a multiple calls",
567		-	root: "a/place",
568		-	mock: indexerMock{
569		-	additionalRoots: map[string][]string{
570		-	"a/place": {
571		-	"another/place",
572		-	"yet-another/place",
573		-	},
574		-	"yet-another/place": {
575		-	"a-quiet-place-2",
576		-	"a-final/place",
577		-	},
578		-	},
579		-	},
580		-	expectedRoots: []string{
581		-	"a/place",
582		-	"another/place",
583		-	"yet-another/place",
584		-	"a-quiet-place-2",
585		-	"a-final/place",
586		-	},
	414	+	expected: errSkipPath,
587	415		},
588	416		}
589		-
590	417		for _, test := range tests {
591	418		t.Run(test.name, func(t *testing.T) {
592		-	assert.NoError(t, indexAllRoots(test.root, test.mock.indexer))
	419	+	assert.Equal(t, test.expected, disallowByFileType("dont/care", test.info, nil))
593	420		})
594	421		}
595	422		}
		skipped 50 lines
646	473		// check that we can access the same file via 2 symlinks
647	474		locations, err = resolver.FilesByGlob("*/link_")
648	475		require.NoError(t, err)
649		-	require.Len(t, locations, 2)
	476	+	require.Len(t, locations, 1) // you would think this is 2, however, they point to the same file, and glob only returns unique files
650	477
651	478		// returned locations can be in any order
652	479		expectedVirtualPaths := []string{
653	480		"link_to_link_to_new_readme",
654		-	"link_to_new_readme",
	481	+	//"link_to_new_readme", // we filter out this one because the first symlink resolves to the same file
655	482		}
656	483
657	484		expectedRealPaths := []string{
		skipped 12 lines
670	497		}
671	498
672	499		func Test_IndexingNestedSymLinks_ignoredIndexes(t *testing.T) {
673		-	filterFn := func(path string, _ os.FileInfo) bool {
674		-	return strings.HasSuffix(path, string(filepath.Separator)+"readme")
	500	+	filterFn := func(path string, _ os.FileInfo, _ error) error {
	501	+	if strings.HasSuffix(path, string(filepath.Separator)+"readme") {
	502	+	return errSkipPath
	503	+	}
	504	+	return nil
675	505		}
676	506
677	507		resolver, err := newDirectoryResolver("./test-fixtures/symlinks-simple", "", filterFn)
		skipped 54 lines
732	562		cwd, err := os.Getwd()
733	563		require.NoError(t, err)
734	564
	565	+	r, err := newDirectoryResolver(".", "")
	566	+	require.NoError(t, err)
	567	+
	568	+	exists, existingPath, err := r.tree.File(file.Path(filepath.Join(cwd, "test-fixtures/image-simple/file-1.txt")))
	569	+	require.True(t, exists)
	570	+	require.NoError(t, err)
	571	+	require.True(t, existingPath.HasReference())
	572	+
735	573		tests := []struct {
736	574		name string
737	575		location Location
		skipped 1 lines
739	577		err bool
740	578		}{
741	579		{
742		-	name: "use file reference for content requests",
743		-	location: NewLocationFromDirectory("some/place", file.Reference{
744		-	RealPath: file.Path(filepath.Join(cwd, "test-fixtures/image-simple/file-1.txt")),
745		-	}),
746		-	expects: "this file has contents",
	580	+	name: "use file reference for content requests",
	581	+	location: NewLocationFromDirectory("some/place", *existingPath.Reference),
	582	+	expects: "this file has contents",
747	583		},
748	584		{
749	585		name: "error on empty file reference",
		skipped 3 lines
753	589		}
754	590		for _, test := range tests {
755	591		t.Run(test.name, func(t *testing.T) {
756		-	r, err := newDirectoryResolver(".", "")
757		-	require.NoError(t, err)
758	592
759	593		actual, err := r.FileContentsByLocation(test.location)
760	594		if test.err {
		skipped 14 lines
775	609		func Test_isUnixSystemRuntimePath(t *testing.T) {
776	610		tests := []struct {
777	611		path string
778		-	expected bool
	612	+	expected error
779	613		}{
780	614		{
781		-	path: "proc/place",
782		-	expected: false,
	615	+	path: "proc/place",
783	616		},
784	617		{
785	618		path: "/proc/place",
786		-	expected: true,
	619	+	expected: fs.SkipDir,
787	620		},
788	621		{
789	622		path: "/proc",
790		-	expected: true,
	623	+	expected: fs.SkipDir,
791	624		},
792	625		{
793		-	path: "/pro/c",
794		-	expected: false,
	626	+	path: "/pro/c",
795	627		},
796	628		{
797		-	path: "/pro",
798		-	expected: false,
	629	+	path: "/pro",
799	630		},
800	631		{
801	632		path: "/dev",
802		-	expected: true,
	633	+	expected: fs.SkipDir,
803	634		},
804	635		{
805	636		path: "/sys",
806		-	expected: true,
	637	+	expected: fs.SkipDir,
807	638		},
808	639		{
809		-	path: "/something/sys",
810		-	expected: false,
	640	+	path: "/something/sys",
811	641		},
812	642		}
813	643		for _, test := range tests {
814	644		t.Run(test.path, func(t *testing.T) {
815		-	assert.Equal(t, test.expected, isUnixSystemRuntimePath(test.path, nil))
	645	+	assert.Equal(t, test.expected, disallowUnixSystemRuntimePath(test.path, nil, nil))
816	646		})
817	647		}
818	648		}
		skipped 5 lines
824	654
825	655		locations, err := resolver.FilesByGlob("**/file.target")
826	656		require.NoError(t, err)
827		-	// Note: I'm not certain that this behavior is correct, but it is not an infinite loop (which is the point of the test)
828		-	// - block/loop0/file.target
829		-	// - devices/loop0/file.target
830		-	// - devices/loop0/subsystem/loop0/file.target
831		-	assert.Len(t, locations, 3)
	657	+
	658	+	require.Len(t, locations, 1)
	659	+	assert.Equal(t, "devices/loop0/file.target", locations[0].RealPath)
832	660		}
833	661
834	662		testWithTimeout(t, 5*time.Second, test)
		skipped 13 lines
848	676		}
849	677		}
850	678
851		-	func Test_IncludeRootPathInIndex(t *testing.T) {
852		-	filterFn := func(path string, _ os.FileInfo) bool {
853		-	return path != "/"
854		-	}
855		-
856		-	resolver, err := newDirectoryResolver("/", "", filterFn)
857		-	require.NoError(t, err)
858		-
859		-	exists, ref, err := resolver.fileTree.File(file.Path("/"))
860		-	require.NoError(t, err)
861		-	require.NotNil(t, ref)
862		-	assert.True(t, exists)
863		-
864		-	_, exists = resolver.metadata[ref.ID()]
865		-	require.True(t, exists)
866		-	}
867		-
868		-	func TestDirectoryResolver_indexPath(t *testing.T) {
869		-	// TODO: Ideally we can use an OS abstraction, which would obviate the need for real FS setup.
870		-	tempFile, err := os.CreateTemp("", "")
871		-	require.NoError(t, err)
872		-
873		-	resolver, err := newDirectoryResolver(tempFile.Name(), "")
874		-	require.NoError(t, err)
875		-
876		-	t.Run("filtering path with nil os.FileInfo", func(t *testing.T) {
877		-	// We use one of these prefixes in order to trigger a pathFilterFn
878		-	filteredPath := unixSystemRuntimePrefixes[0]
879		-
880		-	var fileInfo os.FileInfo = nil
881		-
882		-	assert.NotPanics(t, func() {
883		-	_, err := resolver.indexPath(filteredPath, fileInfo, nil)
884		-	assert.NoError(t, err)
885		-	})
886		-	})
887		-	}
888		-
889	679		func TestDirectoryResolver_FilesByPath_baseRoot(t *testing.T) {
890	680		cases := []struct {
891	681		name string
		skipped 65 lines
957	747		assert.ElementsMatch(t, c.expected, s.List())
958	748		})
959	749		}
	750	+
	751	+	}
	752	+
	753	+	func Test_directoryResolver_resolvesLinks(t *testing.T) {
	754	+	tests := []struct {
	755	+	name string
	756	+	runner func(FileResolver) []Location
	757	+	expected []Location
	758	+	}{
	759	+	{
	760	+	name: "by mimetype",
	761	+	runner: func(resolver FileResolver) []Location {
	762	+	// links should not show up when searching mimetype
	763	+	actualLocations, err := resolver.FilesByMIMEType("text/plain")
	764	+	assert.NoError(t, err)
	765	+	return actualLocations
	766	+	},
	767	+	expected: []Location{
	768	+	{
	769	+	Coordinates: Coordinates{
	770	+	RealPath: "file-1.txt",
	771	+	},
	772	+	//VirtualPath: "file-1.txt",
	773	+	},
	774	+	{
	775	+	Coordinates: Coordinates{
	776	+	RealPath: "file-3.txt",
	777	+	},
	778	+	//VirtualPath: "file-3.txt",
	779	+	},
	780	+	{
	781	+	Coordinates: Coordinates{
	782	+	RealPath: "file-2.txt",
	783	+	},
	784	+	//VirtualPath: "file-2.txt",
	785	+	},
	786	+	{
	787	+	Coordinates: Coordinates{
	788	+	RealPath: "parent/file-4.txt",
	789	+	},
	790	+	//VirtualPath: "parent/file-4.txt",
	791	+	},
	792	+	},
	793	+	},
	794	+	{
	795	+	name: "by glob to links",
	796	+	runner: func(resolver FileResolver) []Location {
	797	+	// links are searched, but resolve to the real files
	798	+	// for that reason we need to place **/ in front (which is not the same for other resolvers)
	799	+	actualLocations, err := resolver.FilesByGlob("*/ink-*")
	800	+	assert.NoError(t, err)
	801	+	return actualLocations
	802	+	},
	803	+	expected: []Location{
	804	+	{
	805	+	Coordinates: Coordinates{
	806	+	RealPath: "file-1.txt",
	807	+	},
	808	+	VirtualPath: "link-1",
	809	+	},
	810	+	{
	811	+	Coordinates: Coordinates{
	812	+	RealPath: "file-2.txt",
	813	+	},
	814	+	VirtualPath: "link-2",
	815	+	},
	816	+	// we already have this real file path via another link, so only one is returned
	817	+	//{
	818	+	// Coordinates: Coordinates{
	819	+	// RealPath: "file-2.txt",
	820	+	// },
	821	+	// VirtualPath: "link-indirect",
	822	+	//},
	823	+	{
	824	+	Coordinates: Coordinates{
	825	+	RealPath: "file-3.txt",
	826	+	},
	827	+	VirtualPath: "link-within",
	828	+	},
	829	+	},
	830	+	},
	831	+	{
	832	+	name: "by basename",
	833	+	runner: func(resolver FileResolver) []Location {
	834	+	// links are searched, but resolve to the real files
	835	+	actualLocations, err := resolver.FilesByGlob("**/file-2.txt")
	836	+	assert.NoError(t, err)
	837	+	return actualLocations
	838	+	},
	839	+	expected: []Location{
	840	+	// this has two copies in the base image, which overwrites the same location
	841	+	{
	842	+	Coordinates: Coordinates{
	843	+	RealPath: "file-2.txt",
	844	+	},
	845	+	//VirtualPath: "file-2.txt",
	846	+	},
	847	+	},
	848	+	},
	849	+	{
	850	+	name: "by basename glob",
	851	+	runner: func(resolver FileResolver) []Location {
	852	+	// links are searched, but resolve to the real files
	853	+	actualLocations, err := resolver.FilesByGlob("**/file-?.txt")
	854	+	assert.NoError(t, err)
	855	+	return actualLocations
	856	+	},
	857	+	expected: []Location{
	858	+	{
	859	+	Coordinates: Coordinates{
	860	+	RealPath: "file-1.txt",
	861	+	},
	862	+	//VirtualPath: "file-1.txt",
	863	+	},
	864	+	{
	865	+	Coordinates: Coordinates{
	866	+	RealPath: "file-2.txt",
	867	+	},
	868	+	//VirtualPath: "file-2.txt",
	869	+	},
	870	+	{
	871	+	Coordinates: Coordinates{
	872	+	RealPath: "file-3.txt",
	873	+	},
	874	+	//VirtualPath: "file-3.txt",
	875	+	},
	876	+	{
	877	+	Coordinates: Coordinates{
	878	+	RealPath: "parent/file-4.txt",
	879	+	},
	880	+	//VirtualPath: "parent/file-4.txt",
	881	+	},
	882	+	},
	883	+	},
	884	+	{
	885	+	name: "by basename glob to links",
	886	+	runner: func(resolver FileResolver) []Location {
	887	+	actualLocations, err := resolver.FilesByGlob("*/link-")
	888	+	assert.NoError(t, err)
	889	+	return actualLocations
	890	+	},
	891	+	expected: []Location{
	892	+	{
	893	+	Coordinates: Coordinates{
	894	+	RealPath: "file-1.txt",
	895	+	},
	896	+	VirtualPath: "link-1",
	897	+	ref: file.Reference{RealPath: "file-1.txt"},
	898	+	},
	899	+	{
	900	+	Coordinates: Coordinates{
	901	+	RealPath: "file-2.txt",
	902	+	},
	903	+	VirtualPath: "link-2",
	904	+	ref: file.Reference{RealPath: "file-2.txt"},
	905	+	},
	906	+	// we already have this real file path via another link, so only one is returned
	907	+	//{
	908	+	// Coordinates: Coordinates{
	909	+	// RealPath: "file-2.txt",
	910	+	// },
	911	+	// VirtualPath: "link-indirect",
	912	+	// ref: file.Reference{RealPath: "file-2.txt"},
	913	+	//},
	914	+	{
	915	+	Coordinates: Coordinates{
	916	+	RealPath: "file-3.txt",
	917	+	},
	918	+	VirtualPath: "link-within",
	919	+	ref: file.Reference{RealPath: "file-3.txt"},
	920	+	},
	921	+	},
	922	+	},
	923	+	{
	924	+	name: "by extension",
	925	+	runner: func(resolver FileResolver) []Location {
	926	+	// links are searched, but resolve to the real files
	927	+	actualLocations, err := resolver.FilesByGlob("*/.txt")
	928	+	assert.NoError(t, err)
	929	+	return actualLocations
	930	+	},
	931	+	expected: []Location{
	932	+	{
	933	+	Coordinates: Coordinates{
	934	+	RealPath: "file-1.txt",
	935	+	},
	936	+	//VirtualPath: "file-1.txt",
	937	+	},
	938	+	{
	939	+	Coordinates: Coordinates{
	940	+	RealPath: "file-2.txt",
	941	+	},
	942	+	//VirtualPath: "file-2.txt",
	943	+	},
	944	+	{
	945	+	Coordinates: Coordinates{
	946	+	RealPath: "file-3.txt",
	947	+	},
	948	+	//VirtualPath: "file-3.txt",
	949	+	},
	950	+	{
	951	+	Coordinates: Coordinates{
	952	+	RealPath: "parent/file-4.txt",
	953	+	},
	954	+	//VirtualPath: "parent/file-4.txt",
	955	+	},
	956	+	},
	957	+	},
	958	+	{
	959	+	name: "by path to degree 1 link",
	960	+	runner: func(resolver FileResolver) []Location {
	961	+	// links resolve to the final file
	962	+	actualLocations, err := resolver.FilesByPath("/link-2")
	963	+	assert.NoError(t, err)
	964	+	return actualLocations
	965	+	},
	966	+	expected: []Location{
	967	+	// we have multiple copies across layers
	968	+	{
	969	+	Coordinates: Coordinates{
	970	+	RealPath: "file-2.txt",
	971	+	},
	972	+	VirtualPath: "link-2",
	973	+	},
	974	+	},
	975	+	},
	976	+	{
	977	+	name: "by path to degree 2 link",
	978	+	runner: func(resolver FileResolver) []Location {
	979	+	// multiple links resolves to the final file
	980	+	actualLocations, err := resolver.FilesByPath("/link-indirect")
	981	+	assert.NoError(t, err)
	982	+	return actualLocations
	983	+	},
	984	+	expected: []Location{
	985	+	// we have multiple copies across layers
	986	+	{
	987	+	Coordinates: Coordinates{
	988	+	RealPath: "file-2.txt",
	989	+	},
	990	+	VirtualPath: "link-indirect",
	991	+	},
	992	+	},
	993	+	},
	994	+	}
	995	+
	996	+	for _, test := range tests {
	997	+	t.Run(test.name, func(t *testing.T) {
	998	+	resolver, err := newDirectoryResolver("./test-fixtures/symlinks-from-image-symlinks-fixture", "")
	999	+	require.NoError(t, err)
	1000	+	assert.NoError(t, err)
	1001	+
	1002	+	actual := test.runner(resolver)
	1003	+
	1004	+	compareLocations(t, test.expected, actual)
	1005	+	})
	1006	+	}
	1007	+	}
	1008	+
	1009	+	func TestDirectoryResolver_DoNotAddVirtualPathsToTree(t *testing.T) {
	1010	+	resolver, err := newDirectoryResolver("./test-fixtures/symlinks-prune-indexing", "")
	1011	+	require.NoError(t, err)
	1012	+
	1013	+	allRealPaths := resolver.tree.AllRealPaths()
	1014	+	pathSet := file.NewPathSet(allRealPaths...)
	1015	+
	1016	+	assert.False(t,
	1017	+	pathSet.Contains("/before-path/file.txt"),
	1018	+	"symlink destinations should only be indexed at their real path, not through their virtual (symlinked) path",
	1019	+	)
	1020	+
	1021	+	assert.False(t,
	1022	+	pathSet.Contains("/a-path/file.txt"),
	1023	+	"symlink destinations should only be indexed at their real path, not through their virtual (symlinked) path",
	1024	+	)
	1025	+
	1026	+	}
	1027	+
	1028	+	func TestDirectoryResolver_FilesContents_errorOnDirRequest(t *testing.T) {
	1029	+	resolver, err := newDirectoryResolver("./test-fixtures/system_paths", "")
	1030	+	assert.NoError(t, err)
	1031	+
	1032	+	var dirLoc *Location
	1033	+	for loc := range resolver.AllLocations() {
	1034	+	entry, err := resolver.index.Get(loc.ref)
	1035	+	require.NoError(t, err)
	1036	+	if entry.Metadata.IsDir {
	1037	+	dirLoc = &loc
	1038	+	break
	1039	+	}
	1040	+	}
	1041	+
	1042	+	require.NotNil(t, dirLoc)
	1043	+
	1044	+	reader, err := resolver.FileContentsByLocation(*dirLoc)
	1045	+	require.Error(t, err)
	1046	+	require.Nil(t, reader)
	1047	+	}
	1048	+
	1049	+	func TestDirectoryResolver_AllLocations(t *testing.T) {
	1050	+	resolver, err := newDirectoryResolver("./test-fixtures/symlinks-from-image-symlinks-fixture", "")
	1051	+	assert.NoError(t, err)
	1052	+
	1053	+	paths := strset.New()
	1054	+	for loc := range resolver.AllLocations() {
	1055	+	if strings.HasPrefix(loc.RealPath, "/") {
	1056	+	// ignore outside of the fixture root for now
	1057	+	continue
	1058	+	}
	1059	+	paths.Add(loc.RealPath)
	1060	+	}
	1061	+	expected := []string{
	1062	+	"file-1.txt",
	1063	+	"file-2.txt",
	1064	+	"file-3.txt",
	1065	+	"link-1",
	1066	+	"link-2",
	1067	+	"link-dead",
	1068	+	"link-indirect",
	1069	+	"link-within",
	1070	+	"parent",
	1071	+	"parent-link",
	1072	+	"parent/file-4.txt",
	1073	+	}
	1074	+
	1075	+	pathsList := paths.List()
	1076	+	sort.Strings(pathsList)
	1077	+
	1078	+	assert.ElementsMatchf(t, expected, pathsList, "expected all paths to be indexed, but found different paths: \n%s", cmp.Diff(expected, paths.List()))
960	1079		}
961	1080

■ ■ ■ ■ ■ ■

syft/source/excluding_file_resolver_test.go

		skipped 55 lines
56	56		resolver := &mockResolver{
57	57		locations: test.locations,
58	58		}
59		-	excludingResolver := NewExcludingResolver(resolver, test.excludeFn)
	59	+	er := NewExcludingResolver(resolver, test.excludeFn)
60	60
61		-	locations, _ := excludingResolver.FilesByPath()
	61	+	locations, _ := er.FilesByPath()
62	62		assert.ElementsMatch(t, locationPaths(locations), test.expected)
63	63
64		-	locations, _ = excludingResolver.FilesByGlob()
	64	+	locations, _ = er.FilesByGlob()
65	65		assert.ElementsMatch(t, locationPaths(locations), test.expected)
66	66
67		-	locations, _ = excludingResolver.FilesByMIMEType()
	67	+	locations, _ = er.FilesByMIMEType()
68	68		assert.ElementsMatch(t, locationPaths(locations), test.expected)
69	69
70	70		locations = []Location{}
71	71
72		-	channel := excludingResolver.AllLocations()
	72	+	channel := er.AllLocations()
73	73		for location := range channel {
74	74		locations = append(locations, location)
75	75		}
		skipped 2 lines
78	78		diff := difference(test.locations, test.expected)
79	79
80	80		for _, path := range diff {
81		-	assert.False(t, excludingResolver.HasPath(path))
82		-	c, err := excludingResolver.FileContentsByLocation(makeLocation(path))
	81	+	assert.False(t, er.HasPath(path))
	82	+	c, err := er.FileContentsByLocation(makeLocation(path))
83	83		assert.Nil(t, c)
84	84		assert.Error(t, err)
85		-	m, err := excludingResolver.FileMetadataByLocation(makeLocation(path))
	85	+	m, err := er.FileMetadataByLocation(makeLocation(path))
86	86		assert.Empty(t, m.LinkDestination)
87	87		assert.Error(t, err)
88		-	l := excludingResolver.RelativeFileByPath(makeLocation(""), path)
	88	+	l := er.RelativeFileByPath(makeLocation(""), path)
89	89		assert.Nil(t, l)
90	90		}
91	91
92	92		for _, path := range test.expected {
93		-	assert.True(t, excludingResolver.HasPath(path))
94		-	c, err := excludingResolver.FileContentsByLocation(makeLocation(path))
	93	+	assert.True(t, er.HasPath(path))
	94	+	c, err := er.FileContentsByLocation(makeLocation(path))
95	95		assert.NotNil(t, c)
96	96		assert.Nil(t, err)
97		-	m, err := excludingResolver.FileMetadataByLocation(makeLocation(path))
	97	+	m, err := er.FileMetadataByLocation(makeLocation(path))
98	98		assert.NotEmpty(t, m.LinkDestination)
99	99		assert.Nil(t, err)
100		-	l := excludingResolver.RelativeFileByPath(makeLocation(""), path)
	100	+	l := er.RelativeFileByPath(makeLocation(""), path)
101	101		assert.NotNil(t, l)
102	102		}
103	103		})
		skipped 69 lines
173	173		}
174	174
175	175		func (r *mockResolver) FilesByMIMEType(_ ...string) ([]Location, error) {
	176	+	return r.getLocations()
	177	+	}
	178	+
	179	+	func (r *mockResolver) FilesByExtension(_ ...string) ([]Location, error) {
	180	+	return r.getLocations()
	181	+	}
	182	+
	183	+	func (r *mockResolver) FilesByBasename(_ ...string) ([]Location, error) {
	184	+	return r.getLocations()
	185	+	}
	186	+
	187	+	func (r *mockResolver) FilesByBasenameGlob(_ ...string) ([]Location, error) {
176	188		return r.getLocations()
177	189		}
178	190
		skipped 20 lines

■ ■ ■ ■ ■ ■

syft/source/file_metadata.go

1	1		package source
2	2
3	3		import (
4		-	"os"
5		-
6	4		"github.com/anchore/stereoscope/pkg/file"
7	5		"github.com/anchore/stereoscope/pkg/image"
8		-	"github.com/anchore/syft/internal/log"
9	6		)
10	7
11		-	type FileMetadata struct {
12		-	Mode os.FileMode
13		-	Type FileType
14		-	UserID int
15		-	GroupID int
16		-	LinkDestination string
17		-	Size int64
18		-	MIMEType string
19		-	}
	8	+	type FileMetadata = file.Metadata
20	9
21		-	func fileMetadataByLocation(img *image.Image, location Location) (FileMetadata, error) {
	10	+	func fileMetadataByLocation(img *image.Image, location Location) (file.Metadata, error) {
22	11		entry, err := img.FileCatalog.Get(location.ref)
23	12		if err != nil {
24	13		return FileMetadata{}, err
25	14		}
26	15
27		-	return FileMetadata{
28		-	Mode: entry.Metadata.Mode,
29		-	Type: newFileTypeFromTarHeaderTypeFlag(entry.Metadata.TypeFlag),
30		-	UserID: entry.Metadata.UserID,
31		-	GroupID: entry.Metadata.GroupID,
32		-	LinkDestination: entry.Metadata.Linkname,
33		-	Size: entry.Metadata.Size,
34		-	MIMEType: entry.Metadata.MIMEType,
35		-	}, nil
36		-	}
37		-
38		-	func fileMetadataFromPath(path string, info os.FileInfo, withMIMEType bool) FileMetadata {
39		-	var mimeType string
40		-	uid, gid := GetXid(info)
41		-
42		-	if withMIMEType {
43		-	f, err := os.Open(path)
44		-	if err != nil {
45		-	// TODO: it may be that the file is inaccessible, however, this is not an error or a warning. In the future we need to track these as known-unknowns
46		-	f = nil
47		-	} else {
48		-	defer func() {
49		-	if err := f.Close(); err != nil {
50		-	log.Warnf("unable to close file while obtaining metadata: %s", path)
51		-	}
52		-	}()
53		-	}
54		-
55		-	mimeType = file.MIMEType(f)
56		-	}
57		-
58		-	return FileMetadata{
59		-	Mode: info.Mode(),
60		-	Type: newFileTypeFromMode(info.Mode()),
61		-	// unsupported across platforms
62		-	UserID: uid,
63		-	GroupID: gid,
64		-	Size: info.Size(),
65		-	MIMEType: mimeType,
66		-	}
	16	+	return entry.Metadata, nil
67	17		}
68	18

■ ■ ■ ■ ■ ■

syft/source/file_metadata_test.go

1	-	//go:build !windows
2	-	// +build !windows
3	-
4	-	package source
5	-
6	-	import (
7	-	"os"
8	-	"testing"
9	-
10	-	"github.com/stretchr/testify/assert"
11	-	"github.com/stretchr/testify/require"
12	-	)
13	-
14	-	func Test_fileMetadataFromPath(t *testing.T) {
15	-
16	-	tests := []struct {
17	-	path string
18	-	withMIMEType bool
19	-	expectedType string
20	-	expectedMIMEType string
21	-	}{
22	-	{
23	-	path: "test-fixtures/symlinks-simple/readme",
24	-	withMIMEType: true,
25	-	expectedType: "RegularFile",
26	-	expectedMIMEType: "text/plain",
27	-	},
28	-	{
29	-	path: "test-fixtures/symlinks-simple/link_to_new_readme",
30	-	withMIMEType: true,
31	-	expectedType: "SymbolicLink",
32	-	expectedMIMEType: "text/plain",
33	-	},
34	-	{
35	-	path: "test-fixtures/symlinks-simple/readme",
36	-	withMIMEType: false,
37	-	expectedType: "RegularFile",
38	-	expectedMIMEType: "",
39	-	},
40	-	{
41	-	path: "test-fixtures/symlinks-simple/link_to_new_readme",
42	-	withMIMEType: false,
43	-	expectedType: "SymbolicLink",
44	-	expectedMIMEType: "",
45	-	},
46	-	}
47	-	for _, test := range tests {
48	-	t.Run(test.path, func(t *testing.T) {
49	-	info, err := os.Lstat(test.path)
50	-	require.NoError(t, err)
51	-
52	-	actual := fileMetadataFromPath(test.path, info, test.withMIMEType)
53	-	assert.Equal(t, test.expectedMIMEType, actual.MIMEType)
54	-	assert.Equal(t, test.expectedType, string(actual.Type))
55	-	})
56	-	}
57	-	}
58	-

■ ■ ■ ■ ■ ■

syft/source/file_resolver.go

		skipped 23 lines
24	24		// FilePathResolver knows how to get a Location for given string paths and globs
25	25		type FilePathResolver interface {
26	26		// HasPath indicates if the given path exists in the underlying source.
	27	+	// The implementation for this may vary, however, generally the following considerations should be made:
	28	+	// - full symlink resolution should be performed on all requests
	29	+	// - returns locations for any file or directory
27	30		HasPath(string) bool
28		-	// FilesByPath fetches a set of file references which have the given path (for an image, there may be multiple matches)
	31	+
	32	+	// FilesByPath fetches a set of file references which have the given path (for an image, there may be multiple matches).
	33	+	// The implementation for this may vary, however, generally the following considerations should be made:
	34	+	// - full symlink resolution should be performed on all requests
	35	+	// - only returns locations to files (NOT directories)
29	36		FilesByPath(paths ...string) ([]Location, error)
30		-	// FilesByGlob fetches a set of file references which the given glob matches
	37	+
	38	+	// FilesByGlob fetches a set of file references for the given glob matches
	39	+	// The implementation for this may vary, however, generally the following considerations should be made:
	40	+	// - full symlink resolution should be performed on all requests
	41	+	// - if multiple paths to the same file are found, the best single match should be returned
	42	+	// - only returns locations to files (NOT directories)
31	43		FilesByGlob(patterns ...string) ([]Location, error)
32		-	// FilesByMIMEType fetches a set of file references which the contents have been classified as one of the given MIME Types
	44	+
	45	+	// FilesByMIMEType fetches a set of file references which the contents have been classified as one of the given MIME Types.
33	46		FilesByMIMEType(types ...string) ([]Location, error)
	47	+
34	48		// RelativeFileByPath fetches a single file at the given path relative to the layer squash of the given reference.
35	49		// This is helpful when attempting to find a file that is in the same layer or lower as another file.
36	50		RelativeFileByPath(_ Location, path string) *Location
37	51		}
38	52
39	53		type FileLocationResolver interface {
	54	+	// AllLocations returns a channel of all file references from the underlying source.
	55	+	// The implementation for this may vary, however, generally the following considerations should be made:
	56	+	// - NO symlink resolution should be performed on results
	57	+	// - returns locations for any file or directory
40	58		AllLocations() <-chan Location
41	59		}
42	60

■ ■ ■ ■ ■ ■

syft/source/file_type.go

1	-	package source
2	-
3	-	import (
4	-	"archive/tar"
5	-	"os"
6	-	)
7	-
8	-	const (
9	-	RegularFile FileType = "RegularFile"
10	-	// IrregularFile is how syft defines files that are neither regular, symbolic or directory.
11	-	// For ref: the seven standard Unix file types are regular, directory, symbolic link,
12	-	// FIFO special, block special, character special, and socket as defined by POSIX.
13	-	IrregularFile FileType = "IrregularFile"
14	-	HardLink FileType = "HardLink"
15	-	SymbolicLink FileType = "SymbolicLink"
16	-	CharacterDevice FileType = "CharacterDevice"
17	-	BlockDevice FileType = "BlockDevice"
18	-	Directory FileType = "Directory"
19	-	FIFONode FileType = "FIFONode"
20	-	Socket FileType = "Socket"
21	-	)
22	-
23	-	type FileType string
24	-
25	-	func newFileTypeFromTarHeaderTypeFlag(flag byte) FileType {
26	-	switch flag {
27	-	case tar.TypeReg, tar.TypeRegA:
28	-	return RegularFile
29	-	case tar.TypeLink:
30	-	return HardLink
31	-	case tar.TypeSymlink:
32	-	return SymbolicLink
33	-	case tar.TypeChar:
34	-	return CharacterDevice
35	-	case tar.TypeBlock:
36	-	return BlockDevice
37	-	case tar.TypeDir:
38	-	return Directory
39	-	case tar.TypeFifo:
40	-	return FIFONode
41	-	}
42	-	return IrregularFile
43	-	}
44	-
45	-	func newFileTypeFromMode(mode os.FileMode) FileType {
46	-	switch {
47	-	case isSet(mode, os.ModeSymlink):
48	-	return SymbolicLink
49	-	case isSet(mode, os.ModeIrregular):
50	-	return IrregularFile
51	-	case isSet(mode, os.ModeCharDevice):
52	-	return CharacterDevice
53	-	case isSet(mode, os.ModeDevice):
54	-	return BlockDevice
55	-	case isSet(mode, os.ModeNamedPipe):
56	-	return FIFONode
57	-	case isSet(mode, os.ModeSocket):
58	-	return Socket
59	-	case mode.IsDir():
60	-	return Directory
61	-	case mode.IsRegular():
62	-	return RegularFile
63	-	default:
64	-	return IrregularFile
65	-	}
66	-	}
67	-
68	-	func isSet(mode, field os.FileMode) bool {
69	-	return mode&field != 0
70	-	}
71	-

■ ■ ■ ■ ■ ■

syft/source/image_all_layers_resolver.go

1	+	package source
2	+
3	+	import (
4	+	"fmt"
5	+	"io"
6	+
7	+	"github.com/anchore/stereoscope/pkg/file"
8	+	"github.com/anchore/stereoscope/pkg/filetree"
9	+	"github.com/anchore/stereoscope/pkg/image"
10	+	"github.com/anchore/syft/internal/log"
11	+	)
12	+
13	+	var _ FileResolver = (*imageAllLayersResolver)(nil)
14	+
15	+	// imageAllLayersResolver implements path and content access for the AllLayers source option for container image data sources.
16	+	type imageAllLayersResolver struct {
17	+	img *image.Image
18	+	layers []int
19	+	}
20	+
21	+	// newAllLayersResolver returns a new resolver from the perspective of all image layers for the given image.
22	+	func newAllLayersResolver(img image.Image) (imageAllLayersResolver, error) {
23	+	if len(img.Layers) == 0 {
24	+	return nil, fmt.Errorf("the image does not contain any layers")
25	+	}
26	+
27	+	var layers = make([]int, 0)
28	+	for idx := range img.Layers {
29	+	layers = append(layers, idx)
30	+	}
31	+	return &imageAllLayersResolver{
32	+	img: img,
33	+	layers: layers,
34	+	}, nil
35	+	}
36	+
37	+	// HasPath indicates if the given path exists in the underlying source.
38	+	func (r *imageAllLayersResolver) HasPath(path string) bool {
39	+	p := file.Path(path)
40	+	for _, layerIdx := range r.layers {
41	+	tree := r.img.Layers[layerIdx].Tree
42	+	if tree.HasPath(p) {
43	+	return true
44	+	}
45	+	}
46	+	return false
47	+	}
48	+
49	+	func (r *imageAllLayersResolver) fileByRef(ref file.Reference, uniqueFileIDs file.ReferenceSet, layerIdx int) ([]file.Reference, error) {
50	+	uniqueFiles := make([]file.Reference, 0)
51	+
52	+	// since there is potentially considerable work for each symlink/hardlink that needs to be resolved, let's check to see if this is a symlink/hardlink first
53	+	entry, err := r.img.FileCatalog.Get(ref)
54	+	if err != nil {
55	+	return nil, fmt.Errorf("unable to fetch metadata (ref=%+v): %w", ref, err)
56	+	}
57	+
58	+	if entry.Metadata.Type == file.TypeHardLink \|\| entry.Metadata.Type == file.TypeSymLink {
59	+	// a link may resolve in this layer or higher, assuming a squashed tree is used to search
60	+	// we should search all possible resolutions within the valid source
61	+	for _, subLayerIdx := range r.layers[layerIdx:] {
62	+	resolvedRef, err := r.img.ResolveLinkByLayerSquash(ref, subLayerIdx)
63	+	if err != nil {
64	+	return nil, fmt.Errorf("failed to resolve link from layer (layer=%d ref=%+v): %w", subLayerIdx, ref, err)
65	+	}
66	+	if resolvedRef.HasReference() && !uniqueFileIDs.Contains(*resolvedRef.Reference) {
67	+	uniqueFileIDs.Add(*resolvedRef.Reference)
68	+	uniqueFiles = append(uniqueFiles, *resolvedRef.Reference)
69	+	}
70	+	}
71	+	} else if !uniqueFileIDs.Contains(ref) {
72	+	uniqueFileIDs.Add(ref)
73	+	uniqueFiles = append(uniqueFiles, ref)
74	+	}
75	+
76	+	return uniqueFiles, nil
77	+	}
78	+
79	+	// FilesByPath returns all file.References that match the given paths from any layer in the image.
80	+	func (r *imageAllLayersResolver) FilesByPath(paths ...string) ([]Location, error) {
81	+	uniqueFileIDs := file.NewFileReferenceSet()
82	+	uniqueLocations := make([]Location, 0)
83	+
84	+	for _, path := range paths {
85	+	for idx, layerIdx := range r.layers {
86	+	ref, err := r.img.Layers[layerIdx].SearchContext.SearchByPath(path, filetree.FollowBasenameLinks, filetree.DoNotFollowDeadBasenameLinks)
87	+	if err != nil {
88	+	return nil, err
89	+	}
90	+	if !ref.HasReference() {
91	+	// no file found, keep looking through layers
92	+	continue
93	+	}
94	+
95	+	// don't consider directories (special case: there is no path information for /)
96	+	if ref.RealPath == "/" {
97	+	continue
98	+	} else if r.img.FileCatalog.Exists(*ref.Reference) {
99	+	metadata, err := r.img.FileCatalog.Get(*ref.Reference)
100	+	if err != nil {
101	+	return nil, fmt.Errorf("unable to get file metadata for path=%q: %w", ref.RealPath, err)
102	+	}
103	+	if metadata.Metadata.IsDir {
104	+	continue
105	+	}
106	+	}
107	+
108	+	results, err := r.fileByRef(*ref.Reference, uniqueFileIDs, idx)
109	+	if err != nil {
110	+	return nil, err
111	+	}
112	+	for _, result := range results {
113	+	uniqueLocations = append(uniqueLocations, NewLocationFromImage(path, result, r.img))
114	+	}
115	+	}
116	+	}
117	+	return uniqueLocations, nil
118	+	}
119	+
120	+	// FilesByGlob returns all file.References that match the given path glob pattern from any layer in the image.
121	+	// nolint:gocognit
122	+	func (r *imageAllLayersResolver) FilesByGlob(patterns ...string) ([]Location, error) {
123	+	uniqueFileIDs := file.NewFileReferenceSet()
124	+	uniqueLocations := make([]Location, 0)
125	+
126	+	for _, pattern := range patterns {
127	+	for idx, layerIdx := range r.layers {
128	+	results, err := r.img.Layers[layerIdx].SquashedSearchContext.SearchByGlob(pattern, filetree.FollowBasenameLinks, filetree.DoNotFollowDeadBasenameLinks)
129	+	if err != nil {
130	+	return nil, fmt.Errorf("failed to resolve files by glob (%s): %w", pattern, err)
131	+	}
132	+
133	+	for _, result := range results {
134	+	if !result.HasReference() {
135	+	continue
136	+	}
137	+	// don't consider directories (special case: there is no path information for /)
138	+	if result.RealPath == "/" {
139	+	continue
140	+	} else if r.img.FileCatalog.Exists(*result.Reference) {
141	+	metadata, err := r.img.FileCatalog.Get(*result.Reference)
142	+	if err != nil {
143	+	return nil, fmt.Errorf("unable to get file metadata for path=%q: %w", result.RequestPath, err)
144	+	}
145	+	// don't consider directories
146	+	if metadata.Metadata.IsDir {
147	+	continue
148	+	}
149	+	}
150	+
151	+	refResults, err := r.fileByRef(*result.Reference, uniqueFileIDs, idx)
152	+	if err != nil {
153	+	return nil, err
154	+	}
155	+	for _, refResult := range refResults {
156	+	uniqueLocations = append(uniqueLocations, NewLocationFromImage(string(result.RequestPath), refResult, r.img))
157	+	}
158	+	}
159	+	}
160	+	}
161	+
162	+	return uniqueLocations, nil
163	+	}
164	+
165	+	// RelativeFileByPath fetches a single file at the given path relative to the layer squash of the given reference.
166	+	// This is helpful when attempting to find a file that is in the same layer or lower as another file.
167	+	func (r imageAllLayersResolver) RelativeFileByPath(location Location, path string) Location {
168	+	layer := r.img.FileCatalog.Layer(location.ref)
169	+
170	+	exists, relativeRef, err := layer.SquashedTree.File(file.Path(path), filetree.FollowBasenameLinks)
171	+	if err != nil {
172	+	log.Errorf("failed to find path=%q in squash: %+w", path, err)
173	+	return nil
174	+	}
175	+	if !exists && !relativeRef.HasReference() {
176	+	return nil
177	+	}
178	+
179	+	relativeLocation := NewLocationFromImage(path, *relativeRef.Reference, r.img)
180	+
181	+	return &relativeLocation
182	+	}
183	+
184	+	// FileContentsByLocation fetches file contents for a single file reference, irregardless of the source layer.
185	+	// If the path does not exist an error is returned.
186	+	func (r *imageAllLayersResolver) FileContentsByLocation(location Location) (io.ReadCloser, error) {
187	+	entry, err := r.img.FileCatalog.Get(location.ref)
188	+	if err != nil {
189	+	return nil, fmt.Errorf("unable to get metadata for path=%q from file catalog: %w", location.RealPath, err)
190	+	}
191	+
192	+	switch entry.Metadata.Type {
193	+	case file.TypeSymLink, file.TypeHardLink:
194	+	// the location we are searching may be a symlink, we should always work with the resolved file
195	+	newLocation := r.RelativeFileByPath(location, location.VirtualPath)
196	+	if newLocation == nil {
197	+	// this is a dead link
198	+	return nil, fmt.Errorf("no contents for location=%q", location.VirtualPath)
199	+	}
200	+	location = *newLocation
201	+	case file.TypeDirectory:
202	+	return nil, fmt.Errorf("cannot read contents of non-file %q", location.ref.RealPath)
203	+	}
204	+
205	+	return r.img.FileContentsByRef(location.ref)
206	+	}
207	+
208	+	func (r *imageAllLayersResolver) FilesByMIMEType(types ...string) ([]Location, error) {
209	+	uniqueFileIDs := file.NewFileReferenceSet()
210	+	uniqueLocations := make([]Location, 0)
211	+
212	+	for idx, layerIdx := range r.layers {
213	+	refs, err := r.img.Layers[layerIdx].SearchContext.SearchByMIMEType(types...)
214	+	if err != nil {
215	+	return nil, err
216	+	}
217	+
218	+	for _, ref := range refs {
219	+	if !ref.HasReference() {
220	+	continue
221	+	}
222	+
223	+	refResults, err := r.fileByRef(*ref.Reference, uniqueFileIDs, idx)
224	+	if err != nil {
225	+	return nil, err
226	+	}
227	+	for _, refResult := range refResults {
228	+	uniqueLocations = append(uniqueLocations, NewLocationFromImage(string(ref.RequestPath), refResult, r.img))
229	+	}
230	+	}
231	+	}
232	+
233	+	return uniqueLocations, nil
234	+	}
235	+
236	+	func (r *imageAllLayersResolver) AllLocations() <-chan Location {
237	+	results := make(chan Location)
238	+	go func() {
239	+	defer close(results)
240	+	for _, layerIdx := range r.layers {
241	+	tree := r.img.Layers[layerIdx].Tree
242	+	for _, ref := range tree.AllFiles(file.AllTypes()...) {
243	+	results <- NewLocationFromImage(string(ref.RealPath), ref, r.img)
244	+	}
245	+	}
246	+	}()
247	+	return results
248	+	}
249	+
250	+	func (r *imageAllLayersResolver) FileMetadataByLocation(location Location) (FileMetadata, error) {
251	+	return fileMetadataByLocation(r.img, location)
252	+	}
253	+

Speed up cataloging by replacing globs searching with index lookups (#1510)