Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Too many branches make branch globs fail #159

Closed
dowobeha opened this issue Nov 7, 2013 · 32 comments
Closed

Too many branches make branch globs fail #159

dowobeha opened this issue Nov 7, 2013 · 32 comments
Assignees
@jhclark
Labels
bug higher priority showstopper
Milestone
V0.4

Comments

@dowobeha
Copy link
Collaborator

dowobeha commented Nov 7, 2013

The following tape fails to run:

task foo
    :: param=(Letter: a b c d e f g h i)
     > letter
{
    echo ${param} > ${letter}
}

task bar
    < letters=$letter@foo[Letter:*]
{
    echo ${letters}
}

However, if the number of branch values are reduced, the tape runs fine:

task foo
    :: param=(Letter: a b c d)
     > letter
{
    echo ${param} > ${letter}
}

task bar
    < letters=$letter@foo[Letter:*]
{
    echo ${letters}
}

Commit d68eadd checks in this broken example as

tutorial/03-07-branch-globbing.tape.BROKEN
@ghost ghost assigned jhclark Nov 7, 2013
dowobeha referenced this issue Nov 7, 2013
@dowobeha
Check in example that breaks branch globs
d68eadd
@NickRuiz
Copy link

Probably repeating the obvious, but explicitly assigning the variables causes the same issue.

task foo
    :: param=(Letter: a b c d e f g)
     > letter
{
    echo ${param} > ${letter}
}

task bar
    < a=$letter@foo[Letter:a] b=$letter@foo[Letter:b] c=$letter@foo[Letter:c]
      d=$letter@foo[Letter:d] e=$letter@foo[Letter:e] f=$letter@foo[Letter:f] 
      g=$letter@foo[Letter:g]
{
    echo ${a} ${b}
}

@jhclark
Copy link
Owner

jhclark commented May 15, 2014

Yep. This one is the primary annoyance of mine right now. The fix is fairly involved and I estimate it at 2-3 weeks worth of work.

Any volunteers to learn some Scala?

Bribe: Fix this one and you get to be a co-author on the paper about the Cool Algorithm that fixes it. I suppose it's not much of a bribe since you will actually be an expert on ducttape by the time you're done. :)

@dowobeha
Copy link
Collaborator Author

Grrr... We have got to get this fixed. This bug should be reclassified as showstopper.

@dowobeha
Copy link
Collaborator Author

@jhclark , what is involved in getting this fixed?

@dowobeha
Copy link
Collaborator Author

@jhclark It's been a year. :-( What is involved in fixing this?

@NickRuiz
Copy link

Seconded. I don't have a lot of time available, but it's painful writing hacks to get around this.

@dowobeha
Copy link
Collaborator Author

dowobeha commented Apr 16, 2017 •
edited
Loading

Interesting update. If the glob is on params rather than on outputs, the slowdown doesn't occur.

task foo
    :: param=(Letter: a b c d e f g h i)
     > letter
{
    echo ${param} > ${letter}
}

task bar
    :: letters=$param@foo[Letter:*]
{
    echo ${letters}
}

@mitchellgordon95
Copy link

mitchellgordon95 commented Jan 23, 2020 •
edited
Loading

Hello from 2020! Want to share my current work-around. Again, this does not work:

ask foo
    :: param=(Letter: a b c d e f g h i)
     > letter
{
    echo ${param} > ${letter}
}

task bar
    < letters=$letter@foo[Letter:*]
{
    echo ${letters}
}

But this does:

task foo
    :: param=(Letter: a b c d e f g h i)
     > letter
{
    echo ${param} > ${letter}
}

task bar1
    < a=$letter@foo[Letter:a]
    < b=$letter@foo[Letter:b]
    < c=$letter@foo[Letter:c]
    < d=$letter@foo[Letter:d]
    > out
{
    cat $a $b $c $d > $out
}

task bar2
    < e=$letter@foo[Letter:e]
    < f=$letter@foo[Letter:f]
    < g=$letter@foo[Letter:g]
    < h=$letter@foo[Letter:h]
    > out
{
    cat $e $f $g $h > $out
}

task bar 
< out1=$out@bar1
< out2=$out@bar2 {
...
}

So I've just been breaking the task that has too many globs into copies with four branches each, and then combining those copies in another task. Obviously this is non-ideal; I'd still like a fix if anyone has it.

Rather than doing the full two-week fix, could we hack on some syntax sugar to make this kind of work-around easier? Also, out of curiosity, what exactly is the problem with globbing? Did the "Cool Algorithm" paper ever get published?

Edit: Sorry, my original example wasn't quite right.

@dowobeha
Copy link
Collaborator Author

Use https://github.com/ExperimentWith/ducttape/files/1725708/ducttape.v0.4.binary.zip

@mitchellgordon95
Copy link

That works, thanks!

Is version 0.4 currently being worked on? I haven't seen any commits on this repo since 2015...

@dowobeha
Copy link
Collaborator Author

Not since 2015. Consider that tarball the 0.4 release.

@jhclark
Copy link
Owner

jhclark commented Jan 25, 2020 via email

@CoderPat
Copy link

Hey @dowobeha , is it possible to upload the code for version 0.4 (or is that in master already)?
I'm thinking of picking up the torch in this project

@CoderPat
Copy link

(or maybe @jhclark knows, sorry for necroing this thread)

@bikestra
Copy link

I dont have the answer but I am so excited that someone might pick up the torch on this awesome project. I am a very happy user of this project and would be happy to contribute ;)

@jhclark
Copy link
Owner

jhclark commented May 17, 2021 via email

@dowobeha
Copy link
Collaborator Author

Hi all! Welcome!

So everything should already be committed, this is just not the correct repo anymore.

All recent development has taken place at https://github.com/ExperimentWith/ducttape. That repo includes a v0.4 tag (https://github.com/ExperimentWith/ducttape/releases/tag/v0.4).

Jon, btw, I'm still interested in the formalism paper, but I've forgotten everything you so painstakingly described to me.

Lane

@dowobeha
Copy link
Collaborator Author

OK. I'm pretty sure the stable branch is where the fixes are. That branch should be considered the primary working branch.

https://github.com/ExperimentWith/ducttape/commits/stable

@dowobeha
Copy link
Collaborator Author

OK. The stable branch is now the primary branch. Also, there is a tech report:

https://github.com/ExperimentWith/ducttape/blob/stable/tech-report/report.tex

@bikestra
Copy link

Thanks for sharing the tech report. Just from reading the code, I had some difficulty understanding how hyperedges and grafts were defined.

@shuoyangd
Copy link

Hi,

Some ducttape users at JHU use issues of this repo to keep track of the pitfalls in ducttape-0.4. Might be useful for future users and developers.

Shuoyang

@CoderPat
Copy link

Wow that is a a lot of new info, glad I necro-ed this issue. I'll read the tech report thanks
Regarding what I plan to do with this @jhclark , I would probably start slow, adding a couple of features that I feel are missing (branch aggregation based only the branches on plan rather than all), or fixing some annoying things (ducttape seems to freeze if you re-order a branch).
But ye I think this project is a undervalued gem and eventually I would like to port it to python (albeit that is a big endeouver)!
Given this, it would be better to probably fork from https://github.com/ExperimentWith/ducttape/commits/stable right?

@jhclark
Copy link
Owner

jhclark commented May 18, 2021 via email

@bikestra
Copy link

branch aggregation based only the branches on plan rather than all

Does anyone have a good idea on how this could be implemented? Currently I have a gigantic tape file with a lot of branches, but each plan utilizes only few of them. I need to wait for like 10 mins every time I run ducttape with this tape file, so I was hoping to improve the speed of iteration.

@CoderPat
Copy link

I need to wait for like 10 mins every time I run ducttape with this tape file, so I was hoping to improve the speed of iteration.

What takes 10m? The actual loading of the workflow or running the workflow for all branches?
If the latter (due to a branch aggregation) I would suggest temporarily removing branch values every time you run a plan.
For the former, I suggest filing a bug/issue here https://github.com/CoderPat/ducttape/tree/revive-ducttape .
I'll try to have a version 0.5 ready by next week and I'll start working on improvements

@bikestra
Copy link

bikestra commented May 27, 2021 via email

@CoderPat
Copy link

CoderPat commented Oct 8, 2021

Hey sorry for necroing again, but after a delay due to PhD life, version 0.5 is (finally) released!
https://github.com/CoderPat/ducttape/releases/tag/v0.5
It was mostly a quality-of-life update, but ye I have a couple of features I want in the future

@dowobeha
Copy link
Collaborator Author

dowobeha commented Oct 8, 2021 via email

@shuoyangd
Copy link

Nice! Do we have proper branch globbing now? Or is this just changing build configs?

@CoderPat
Copy link

CoderPat commented Oct 8, 2021

@dowobeha ye it's in the release. I'll copy them below

  • Update scala version to 2.12
  • Updated dependencies needed to support 2.12
  • Change build system to use SBT everywhere. This includes dependencies and packaging
  • Change CI to GitHub actions
  • Add Slurm submitter as a builtin
  • Add Visual Studio Code extension for syntax highlighting

I'll be happy to merge to upstream. But I'll probably maintain my fork since admin rights allows me more fine control over CI and other stuff (unless you don't mind giving me admin/manager access to upstream)

@shuoyangd what do you mean by proper branch globbing? If you mean the slow down when branches increase, that was been fixed since 0.4

@shuoyangd
Copy link

My bad -- I was thinking about this issue when I saw the title.

I also don't believe that the slowness has been completely fixed since 0.4. Your comment on May 27th was also suggesting temporarily removing branches when things slow down.

@CoderPat
Copy link

CoderPat commented Oct 8, 2021

ahhh sorry I was misunderstanding the problem
Your problem is branch globing affecting all possible branches rather only the ones on the realization right?
Ye this is still not "fixed" (albeit I wouldn't call it broken since I think that is just the expected behaviour).
I think a solution would involve including a new globbing operator that acts only on the branches in a realization (albeit I think this will be tricky in terms of implementation).
Nevertheless I think this is important feature since I have the same problem alot of times and I'll add it as an issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jhclark jhclark

Labels
bug higher priority showstopper
Projects
None yet
Milestone
V0.4
Development

No branches or pull requests
7 participants
@jhclark @dowobeha @NickRuiz @bikestra @shuoyangd @mitchellgordon95 @CoderPat