Multiples input for plugins and concurrency

[acarnage]

Hey ! I'm reposting my question here for more visibility :)

I'm trying to build a first plugin basically launching "dirb" against an URL.
I wonder how i can write the hcl code in order to target 2 urls. My following attempts fail with "index out of range" (note that the plugin code works perfectly fine with one URL)

resource "dirb" "dirb" {
for_each = ["https://www.google.fr","https://www.google.com"] 
target="${each.key}"
}

Moreover, i think it would be interesting to add a "concurrency" feature. During a pentest engagement there are multiples IPs to target so i would be awesome to launch a decker instance for each target IP especially if time consuming tasks are involved (as dirb :) ).

[stevenaldinger]

I'm going to try to reproduce this tonight, I have no idea why that wouldn't work but I'll get back to you soon.

As far as concurrency goes, that's something I'm really interested in as well. "Background processes" / event driven configs were the other thing I'd really like to figure out. Use case would be constantly monitoring for wifi/bluetooth clients near by and launching a decker pipeline against them as they showed up. If you have thoughts on exactly how to implement either (or both at the same time, they might be a bit related) I'd love to hear you out. Studying drone (ci/cd - drone.io) is probably where I'll start for research, they've got a clever team and have tackled similar problems. I've been consumed with career work lately and haven't had much time to work on decker unfortunately.

In your case since all the values will be known ahead of time, it might make sense to have the plugin accept an array instead of a single value, and the plugin could handle the concurrency for now if more than one value is passed in. I'll definitely keep you up to date on any concurrency efforts.

[acarnage]

In your case since all the values will be known ahead of time, it might make sense to have the plugin accept an array instead of a single value, and the plugin could handle the concurrency for now if more than one value is passed in.

I kinda agree with you if we can fix the problem I illustrated :). But I would say that it must be more robust if the plugins stay simpler. I would say that a plugin should handle concurrency for the given host/target.

Let's say a target has 2 webservices open, the dirb plugin should be able to be concurrent for both services. But in another way, i think that having the ability to run the same config against several host concurrently will be very interesting.

The first thing that comes to my mind would be a go script that run multiples decker instance for a given target list. But it is surely not the more robust implementation.

EDIT :
Maybe having some kind of a wrapper above the decker functions would allow to manage this more properly. Just having some kind of a function taking as inputs variables and a config file and launching decker could be interesting.

[stevenaldinger]

I kinda agree with you if we can fix the problem I illustrated

^ Do you have that same problem when you switch the dirb.hcl file's target type to list and pass in the array as target instead of a for_each?

... go script that run multiples decker instance for a given target list... Just having some kind of a function taking as inputs variables and a config file and launching decker could be interesting.

Those are interesting thoughts for sure. I wanted the single decker process to be able to handle it originally but then the problem is how to handle things after the concurrency. Does everything for that plugin run concurrently and then join back together for the output, or after the concurrency does everything that references the output continue to run concurrently for each input in their own lanes? I think that's one of the things drone ran into originally too. I'm hesitant to start coding without case studies but I really would love to tackle that problem.

I actually have a "distributed" version of decker too that has a browser UI and launches decker instances in k8s or individual docker containers that's meant to handle some concurrency and routine "cron" type situations. It was meant to be a potential business attempt/proprietary project but I'm not sure I'll realistically ever get to that, if it interests you at all I'd be happy to share the work.

[acarnage]

^ Do you have that same problem when you switch the dirb.hcl file's target type to list and pass in the array as target instead of a for_each?

I'm trying to test this. But a List decoder must be implemented. I tried the following code but it fails. I am not very familiar to golang and so with gocty

Code tested:
` func (*Decoder) GetList(ctyVal cty.Value) []string {
var val []string
err := gocty.FromCtyValue(ctyVal, &val)

if err != nil {
	fmt.Println("CTY VAL:", ctyVal)
	fmt.Println("Error decoding string:", err)
}

return val

} `

Error returned while testing with a plugin :
root@3436f849578a:/go/src/github.com/stevenaldinger/decker# ./decker examples/hello-world.hcl CTY VAL: {{{{} [{{{} 83}} {{{} 83}}]}} [http://google.fr http://google.com]} Error decoding string: list or set value is required [] DECKER: Ran plugin 1 of 1: dirb (dirb) root@3436f849578a:/go/src/github.com/ste

I would be please to give a look to your distriuted version. It indeed may answer the need of concurrency by providing several decker instances

[acarnage]

Edit :

I managed to solve the problem :

func (*Decoder) GetList(ctyVal cty.Value) []string {
	var res []string
	var decodedVal string
	vals := ctyVal.AsValueSlice()
	for _, val := range vals {
		err := gocty.FromCtyValue(val, &decodedVal)
		if err != nil {
			fmt.Println("CTY VAL:", ctyVal)
			fmt.Println("Error decoding string:", err)
		} else {
			res = append(res, decodedVal)
		}

	}

	return res
}

I can now perform a dirb against a target list as input !

[stevenaldinger]

I finally have time to work on this project again, working on a massive refactor/rewrite that's a lot more tested and stable. Thanks for being one of the first to really try it. If you had any thoughts you remember on parts that could be improved/features you would've liked, please let me know. Stability/maintainability are priority this time around but database storage and concurrency are high on the list too