Provide the support for ALTER Command

[abhiitsaboutme]

Hi,

The current MQ Plugin only supports DEFINE and DELETE option. But there are many cases where we want to change the property of Queue and this could be easily done using ALTER command.

If we have to define the new Queue, we can use the following command.
DEFINE QLOCAL('TEST.1.TEST01.IN') + DEFSOPT(SHARED) + DESCR('TEST QUEUE') + MAXDEPTH(10000) + MAXMSGL(3100) + NOTRIGGER + TRIGTYPE(NONE)

Similarly if I want to change only 1 attribute example MAXDEPTH, I can execute the below sample ALTER command and it would do the work:

ALTER QLOCAL('TEST.1.TEST01.IN') + MAXDEPTH(10000)

For this similar ALTER command to work, we need to make changes in Groovy code and add the ALTER command as the valid user option.

Could you please check the feasibility as this would really help lot of projects who do not want to DELETE and re define the queue just to make small change which can be easily achieved by using ALTER command.

[reachsenthilnathan]

@mayur-raja please see the comment above.

[abhiitsaboutme]

Thanks @reachsenthilnathan - ALTER option will help everyone as we can change Queue properties without any downtime.

[abhiitsaboutme]

In MQSCUtil.groovy we have array defined for valid command options

def supportedCommands = ['DEFINE', 'DELETE'] as String[]

[mayur-raja]

@abhiitsaboutme Yes, I agree that it would be useful to add ALTER support but unfortunatey, it is not just a case of adding the ALTER command to the list of supported commands. We need to code for the subtle differences in the arguments that can be ALTERed. This is because not all attributes that can be specified on a DEFINE command are ALTERable. Also, this would need to be considered for each type of Queue resource that ALTER is to be supported for.

@reachsenthilnathan I am rather busy with other work this week but I'll have a detailed look at this next week and get back to you.

Thank you.

[abhiitsaboutme]

Hi @mayur-raja - We have similar situation with DELETE also. If you look into DELETE, user has to specify the attributes in JSON, if we just change from DEFINE to DELETE in normal json file then during execution via CSQUTIL - DELETE statement fails as it accepts only Queue name.

So for ALTER it can be solved in same way that we have to consider the valid attributes in JSON file.

[reachsenthilnathan]

Hi @mayur-raja we can take this like our regular RFEs. Please do your initial analysis to check for feasibility. Then we can work together to see how/when we want to implement it.

Thanks @abhiitsaboutme for your suggestion to enhance the plugin.

[mayur-raja]

@reachsenthilnathan Yes, I agree, RFE is the best approach. Sure, I'll drop you a note later next week. Thank you.

[abhiitsaboutme]

Hello,

Thanks so much for considering the request. Also, I would like to highlight the correction needed for DELETE.

If the user wants to DELETE a QUEUE and the JSON file contains all the attributes related to Queue then we have an issue.

DELETE QLOCAL('QUEUE.1.TEST00.IN') +
DEFSOPT(SHARED) +
DESCR('Queue Created from UCD for Test') +
MAXDEPTH(10000) +
MAXMSGL(3100) +
NOTRIGGER +
TRIGTYPE(NONE)
CSQN205I COUNT= 8, RETURN=00000008, REASON=FFFFFFFF
CSQ9001E (MQ 'DEFSOPT' is invalid
CSQ9001E (MQ 'DESCR' is invalid
CSQ9001E (MQ 'MAXDEPTH' is invalid
CSQ9001E (MQ 'MAXMSGL' is invalid
CSQ9001E (MQ 'NOTRIGGER' is invalid
CSQ9001E (MQ 'TRIGTYPE' is invalid
CSQ9023E (MQ CSQ9SCND 'DELETE QLOCAL' ABNORMAL COMPLETION
CSQU057I 1 commands read
CSQU058I 1 commands issued and responses received, 1 failed
CSQU143I 1 COMMAND statements attempted
CSQU144I 1 statements executed successfully
CSQU148I CSQUTIL Utility completed, return code=0
1 job(s) completed.

Here you can see DELETE contains different arguments like MAXDEPTH, MAXMSGL, DESCR - all these are not needed. The DELETION fails if we give all arguments,

So if we change the plugin code and capture only DELETE along with Queue name then we can solve this issue also. This way user don't have to change JSON file based on the action.

[abhiitsaboutme]

Hi @reachsenthilnathan ,

Any updates on the issue?

[mayur-raja]

Hi guys, apologies but a colleague reminded me that with the DEFINE command, you can also do a DEFINE with the REPLACE option. This behaves like an ALTER command. This should avoid the need to DELETE and re-DEFINE the queue.

Also, thank you for the further details about the issue with the DELETE command. It should not be working the way you describe. For DELETE, you should only need to specify the minimum set of attributes as defined by MQ. We will double check that and get back to you in a few days.

Regards .. Mayur

[mayur-raja]

@abhiitsaboutme As for supporting an actual ALTER command, @reachsenthilnathan and I have recently exchanged emails about this and @reachsenthilnathan will provide an update in due course.

Thank You.

[abhiitsaboutme]

HI @reachsenthilnathan, Hi @mayur-raja,
Thanks for the support. Please check it soon as customer is really pushing us for this change and we want to go live as soon as possible

[reachsenthilnathan]

@abhiitsaboutme Supporting the actual ALTER command is a RFE. Please raise it in the new RFE portal, https://cloud-platform.ideas.aha.io/ideas/. Since DEFINE with REPLACE is similar to ALTER, that would be our preferred way. Can you see whether DEFINE with REPLACE solves your problem with ALTER?

On the DELETE part, we are still investigating whether that is a problem. But even there with a separate JSON file based on the action, DELETE works, right?

[abhiitsaboutme]

Hi @reachsenthilnathan ,
I tested and REPLACE options works fine for our case. I need now changes for DELETE then we are good to go ahead for our plugin changes.

For Delete we need the Queue name and CMDSCOPE to specify Queue manager. I have already tested it and it works fine

[reachsenthilnathan]

Thanks @abhiitsaboutme

Here you can see DELETE contains different arguments like MAXDEPTH, MAXMSGL, DESCR - all these are not needed. The DELETION fails if we give all arguments,

So this is the only problem now, correct? Here if your DELETEhas more parameters, it fails. And you are asking us to ignore the additional parameters, as it is easy for whoever is generating these files to generate the same pattern for DEFINE and DELETE? Is my understanding, correct?

[abhiitsaboutme]

Hi @reachsenthilnathan ,
Yes, right.
What we do is we keep the same json file with all attributes and its value in it. But if the queue needs to be deleted we just change the command from DEFINE to DELETE and it gets deleted.

But as per the current plugin design it picks all attributes for DELETE also and submits JCL. There we face issue.

In our changed plugin version, we removed all attributes during the CSQUTIL command creation and keep only Queue name and CMDSCOPE and this works perfectly fine during execution.

[mayur-raja]

Hi Guys,
I remembered that when we wrote the plugin, we had decided that the plugin should not perform validation of the MQ fields. Validation should instead be left to the Queue Manager to perform (since the MQSC support provided by IBM MQ already performs full validation). So, @reachsenthilnathan, this means that you should be able to add the ALTER command to the list of commands supported by MQSCUtil.groovy. And, once that has been added, @abhiitsaboutme, so long as you ensure that any command (DEFINE, DELETE and ALTER) that you specify in the triplet of files (in JSON format) only contains those attributes that are supported by the MQ command (as coded in the the plugin file: IBM-MQ-zOS-UCD/src/main/zip/resourceMappings/mqResourceAttributes.mappings and in the description of the MQSC command in the IBM MQ Knowledge Center), it should work. However, you should note that the plugin only supports those attributes that were supported by IBM MQ at the time of writing the plugin. Since then, a few new attributes have been introduced which are not yet supported by the plugin.

In fact, if you want to try the ALTER command out, you could add it to the list of commands in your copy of the MQSCUtil.groovy file to confirm if it works or not. Trying changing:

def supportedCommands = ['DEFINE', 'DELETE'] as String[]

to

def supportedCommands = ['DEFINE', 'DELETE', 'ALTER'] as String[]

and please let us know if it works or not. Also note that it would not be wise to add the DISPLAY command to the list of supported commands because the IBM-MQ-zOS-UCD/src/main/zip/resourceMappings/mqResourceAttributes.mappings file currently does not support runtime properties like CURDEPTH, IPPROCS, IPPROCS, etc.

Thank you.

Regards .. Mayur

[mayur-raja]

Hi @reachsenthilnathan ,
Yes, right.
What we do is we keep the same json file with all attributes and its value in it. But if the queue needs to be deleted we just change the command from DEFINE to DELETE and it gets deleted.

But as per the current plugin design it picks all attributes for DELETE also and submits JCL. There we face issue.

In our changed plugin version, we removed all attributes during the CSQUTIL command creation and keep only Queue name and CMDSCOPE and this works perfectly fine during execution.

So guys, as stated in my previous update, we made a conscious decision not to validate and only include properties that were valid for a given command. Yes, I agree that this could be done, and to some degree enhances the plugin, but given that IBM MQ administrators should be familiar with the MQSC commands and the properties that could be supplied on each MQSC command, we decided that administrators should know which properties to specify in the triplet of JSON files.

I'm just letting you know what we had decided at the time of writing the plugin.

Regards .. Mayur

[abhiitsaboutme]

Hi @mayur-raja,

Thanks for the details information. But as discussed with Senthil I am looking support for DELETE command as it doesnt make any sense to have DELETE when it cannot work. And its not about validation but having the basic command work.

Also, we dont need support for DISPLAY as we dont used UCD for displaying Queue properties.

[mayur-raja]

Hi @abhiitsaboutme,

I don't believe the DELETE command does not work. It is not supported in the way you are trying to use it. By simply replacing the command from DEFINE to DELETE, you are propagating the attributes of the DEFINE command with the DELETE command. If you had explicitly only specified those attributes that are supported by the DELETE command, the DELETE command would have worked. This is how other customers are using it.

The plugin could strip off DEFINE command properties when submitting the DELETE command and only leave the queue name and command scope, but it is not ideal as while this is what you may want and it works for most queue deletes, it may not be what other customers want and the plugin design needs to be generic. Users need to also be able to specify PURGE/NOPURGE, QSGDISP, AUTHREC, etc. properties on the DELETE command.

I suppose the plugin could do a check to (if specified) only include those properties that are supported by each command but then you are adding more knowledge about MQ behaviour into the plugin. And of course this will need to be kept in sync. with IBM MQ in the future.

I'll leave it to @reachsenthilnathan to decide what he wants to support.

Regards .. Mayur

[reachsenthilnathan]

Thanks @mayur-raja . I agree with you that plugin needs to have very little business logic. More we load into the plugin, it becomes difficult for customers to customize the input files for their needs.

And it is difficult to get all the permutations and combinations right. So we would like to stick to the following statement at this point:

we made a conscious decision not to validate and only include properties that were valid for a given command

Plugin is still open source and if customer wants to change it for their need, they can. But we would like to keep our support the plugin in this site. And this plugin will always expect the customer to pass the right parameters.

[reachsenthilnathan]

@shabaz213 fyi

[mayur-raja]

Hi @abhiitsaboutme, yesterday you posted an append requesting an example of coding the replace option. I do not see your append anymore but here is an example. Incidentally. in case you did not know, we describe the syntax of the json structure in the white paper that is shipped as part of the doc. for the plugin.

The following example also shows how you would code a delete queue request (apologies but this tool seems to lose the padding so makes the example a little unreadable. However, I hope you can follow it):

{
"resource": {
"queue": [{
"command": "DEFINE",
"name": “TEST.QA”,
"type": "alias",
"optionalParms": {
"commandScope": "",
“replace": “replace"
},
"alias": {
"targetName": “TEST",
"targetType": "queue"
}
},
{
"command": "DELETE",
"name": “TEST.QL”,
"type": “local”
}],
}
}

Regards .. Mayur

[abhiitsaboutme]

Hi @mayur-raja ,
Thanks for the information. I had old documentation and in that it was missing the information for replace. But later I found the new one and it has example as you mentioned so it worked fine.

That's why I removed my comment.

Anyways thanks for sharing the example :-)

[mayur-raja]

Hi @abhiitsaboutme, Ah .. nps, glad you found the new doc.

Regards .. Mayur

[abhiitsaboutme]

Hi @mayur-raja ,

One more question - can we upload a .tar or zip file containing all 3 files for a queue and then unzip it. Will the plugin works?

[reachsenthilnathan]

This plugin cannot unzip it. But if you can unzip it in a shell step before and have it in the right directories, this plugin should work.

[mayur-raja]

@abhiitsaboutme and @reachsenthilnathan I've not tried this, but you could try to add a UCD File Utils plugin step to unzip zipped files and store them in the UCD repository. Then you can drive the Download Artifacts step to download the triplet of files as required by the MQSCUtil plugin.

[abhiitsaboutme]

Hi @reachsenthilnathan and @mayur-raja ,
Thanks for the hint and suggestion. It worked fine. I have used the shell to untar all the tar files which are downloaded and the put the 3 mq files for each queue in specific folder. Once the CSQUTIIL job is executed, we delete the tar folder also using shell.

This worked perfectly fine.

[mayur-raja]

@abhiitsaboutme Awesome ! Glad to hear that ! And thanks to @reachsenthilnathan for the suggestion to use the shell !

[abhiitsaboutme]

Hi @mayur-raja ,
Yes, Thanks to both of you. Shell was better as I could write a generic script which is independent of the tar name and does the required work.

[mayur-raja]

@abhiitsaboutme YW, if you can share a generic/anonymized version of the shell script here then other folks could also benefit from this in the future. But, if you cannot, maybe for confidentiality reasons, then I understand. Thank you.

[abhiitsaboutme]

Hi @mayur-raja ,
Sure, I can share the script, it has nothing confidential in it.

In the below shell script we create folder with the application process id name so that its unique. Then I

Following steps are done in below shell script

• Create new subfolder with the application process id name so that its unique for every request.
• Copy all Tar from the downloaded path to the newly created process-id subfolder.
• Remove tars from the main folder after they have been copied to new subfolder
• Untar the all tar files
• Copy the content from the tar folders to process-id folder
• remove tar files
• remove the subfolder created for every tar during untar process

#!/dc/tools/bin/bash -xe
echo "============="
echo ${p:applicationProcess.id}
processid=echo ${p:applicationProcess.id}
echo "============="
ls -ltr
echo "===2========="
mkdir $processid
for filename in *.tar
do
cp $filename $processid/$filename
rm -r $filename
done
cd $processid
for filename in .tar
do
tar -xovf $filename
tarname=echo $filename| cut -f 1 -d '.'
mv $tarname/ .
rm -r $tarname
rm -r $filename
done
echo "===3========="
ls -ltr

And in the Generate MQSC Command plugin I have mentioned "${p:applicationProcess.id}" in the File Sub Folders field so during execution it checks the MQ JSON files at the given location.

This way for every deployment we have a unique folder. If the user wants to clean these folder they can execute another shell script at the end and remove the newly created subfolder with process name.

[abhiitsaboutme]

It can be optimized easily, for the time being I used this to check and do tracing.

[mayur-raja]

@abhiitsaboutme Many thanks for sharing the shell script. @reachsenthilnathan You may want to consider adding a section in the white paper to briefly document how to do this. Then you can include the script and give credit to @abhiitsaboutme. I think it would be a useful thing to capture for future users of the plugin. Thank you.