Adds var<mat> specializations for lower/upper bounds

[SteveBronder]

Summary

This adds specializations for lower / upper / lower_upper constraints to work with var. I had to do a few other things for this to work.

1. To test this I had to write a specialization for [lb/up/lub]_constrain that worked for Matrix<var> and var_value<Eigen::Matrix>. That means that a few things like check_less etc have to now take / iterate over general matrices.

2. I had to fix elt_multiply() to allow for multiplying a scalar and matrix

3. I'm not sure how to handle the negative and positive infinity checks for the matrix versions. Previously this was checked on a per coefficient basis. But now if you have a vector representing the bounds for a vector, if any of them are +/- infinity then it returns back the input vector without the transformation. I feel like I'm not understanding how this is used. If a vector represents the bound then will it ever be expected that some of those values in the vector would be infinite and some would not be? If that's the case then I need to write specific var specializations which isn't that bad but it would be nice to not have to.

Tests

Tests added for matrix versions. It looks like lub_constrain() has not tests? So I added ones for testing those as well.

./runTests.py -j2 ./test/unit/math/mix/fun/lub_constrain_test.cpp  ./test/unit/math/mix/fun/lb_constrain_test.cpp  ./test/unit/math/mix/fun/ub_constrain_test.cpp 

Side Effects

Yes see (3) above

Release notes

Adds bound constraint functions for var<matrix> functions

Checklist

• Math issue How to add static matrix? #1805

• Copyright holder: Steve Bronder

  The copyright holder is typically you or your assignee, such as a university or company. By submitting this pull request, the copyright holder is agreeing to the license the submitted work under the following licenses:
  - Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
  - Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

• the basic tests are passing

  • unit tests pass (to run, use: ./runTests.py test/unit)
  • header checks pass, (make test-headers)
  • dependencies checks pass, (make test-math-dependencies)
  • docs build, (make doxygen)
  • code passes the built in C++ standards checks (make cpplint)

• the code is written in idiomatic C++ and changes are documented in the doxygen

• the new changes are tested

[bbbales2]

I had to fix elt_multiply() to allow for multiplying a scalar and matrix

Nice. This will make a placeholder fma easy to write if we still need that (which I have been avoiding).

If a vector represents the bound then will it ever be expected that some of those values in the vector would be infinite and some would not be?

I think that makes sense. Some variables constrained and some not. Will have to check the code to make sure I'm reading that right tho.

[bbbales2]

Had a closer look. You're right that the ifs are handling infinities incorrectly.

I think we'll just have to write autodiff for each of these things manually.

Since these constraints are only eigen types and this is only a few functions, this shouldn't be too hard to do. I can do it tomorrow if you've got other stuff to keep you busy. If we do this we can also ditch the new infinity checks unless you want to keep them.

[bbbales2]

@SteveBronder I removed the infinite checks and cleaned this up.

I thought a little more about #2291 -- this already wasn't working right for vector constraints so I'm not gonna worry about supporting this in the future.

The only case this worked was if someone had a scalar constraint that was a infinite, and so I added failing error checks for that. It is hopefully not more expensive than the infinite checks that were already there.

Should be ready to review if tests pass.

[SteveBronder]

Just added some stuff that went a little loco with the moves. But if we need to move things sometimes and we need to use to_ref then I think we need to detect when to_ref gives us back an rvalue that needs to be moved

[SteveBronder]

The other issue is we can't do stuff like

  return make_holder([&x](const auto& lb_ref) { return add(exp(x), lb_ref); },
                     lb_ref);

Since x could fall out of scope we need to pass x in as an argument to make_holder

  return make_holder([](auto& lb_ref, const auto& xx) { return add(exp(xx), lb_ref); },
                     lb_ref, x);

[bbbales2]

I don't think we need to forward through to_ref. If the to_ref doesn't need to eval, it's cheap.

Since x could fall out of scope

My assumption is that input arguments don't need held, but the expression semantics aren't clear to me (when we need holder and when we don't).

[bbbales2]

As an example of not holding inputs: https://github.com/stan-dev/math/blob/develop/stan/math/prim/fun/transpose.hpp

Edit: So I think we're making the assumption that inputs don't need held, since certainly if they did then we'd need a holder there.

[bbbales2]

I think this and stan-dev/stan#2995 are ready to go. @SteveBronder you good with this? If so @rok-cesnovar can do the mergin'.

[SteveBronder]

Yep!

[rok-cesnovar]

Oh, this two PRs need to go in in-sync?

[rok-cesnovar]

Restarted with proper branches, please approve both and I can handle merging this once it passes.

[SteveBronder]

Oh, this two PRs need to go in in-sync?

Yep!

Thanks will do!

[stan-buildbot]

---

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
gp_pois_regr/gp_pois_regr.stan	3.42	3.47	0.99	-1.46% slower
low_dim_corr_gauss/low_dim_corr_gauss.stan	0.02	0.02	0.97	-3.49% slower
eight_schools/eight_schools.stan	0.11	0.11	1.0	-0.35% slower
gp_regr/gp_regr.stan	0.15	0.15	1.02	1.65% faster
irt_2pl/irt_2pl.stan	5.32	5.27	1.01	0.96% faster
performance.compilation	90.81	88.38	1.03	2.68% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	8.61	7.8	1.1	9.38% faster
pkpd/one_comp_mm_elim_abs.stan	28.4	28.04	1.01	1.24% faster
sir/sir.stan	134.83	131.43	1.03	2.52% faster
gp_regr/gen_gp_data.stan	0.04	0.05	0.96	-4.65% slower
low_dim_gauss_mix/low_dim_gauss_mix.stan	3.04	3.01	1.01	1.12% faster
pkpd/sim_one_comp_mm_elim_abs.stan	0.39	0.38	1.04	3.71% faster
arK/arK.stan	2.56	2.52	1.02	1.59% faster
arma/arma.stan	0.61	0.73	0.83	-20.51% slower
garch/garch.stan	0.53	0.66	0.81	-23.32% slower
Mean result: 0.987123003819

---

Jenkins Console Log
Blue Ocean
Commit hash: 8d62db852c55835c5f8b08ea97a2d345b5494f00

---

Machine information

ProductName: Mac OS X ProductVersion: 10.11.6 BuildVersion: 15G22010

CPU:
Intel(R) Xeon(R) CPU E5-1680 v2 @ 3.00GHz

G++:
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 7.0.2 (clang-700.1.81)
Target: x86_64-apple-darwin15.6.0
Thread model: posix

Clang:
Apple LLVM version 7.0.2 (clang-700.1.81)
Target: x86_64-apple-darwin15.6.0
Thread model: posix

[SteveBronder]

Nice! @bbbales2 if you hit the approve button I think we are good to go!

@rok-cesnovar I think this was the last piece we needed for the stance impl! First thing we need to do is use the matrix specializations for the constraints then we can add varmat!