Stuff I'm Up To

Technical Ramblings

JIRA, Confluence and Nginx — September 15, 2018

JIRA, Confluence and Nginx

With Atlassian Jira Software and Confluence installed onto the same server I thought I’d investigate setting things up so we don’t have to use the default TCP port type of access over HTTP. instead let’s setup a reverse proxy using HTTPS over TCP 443 that forwards to the TCP 8080 and 8090 ports.

The aim is to get Jira accessible as https://jira.domain.local and Confluence as https://jira.domain.local/confluence.

Continue reading

Advertisements
JIRA Software and Confluence — September 14, 2018

JIRA Software and Confluence

Installing Atlassian Jira Software onto an in-house or self-hosted server is as simple as following the Jira installation guide. The only thing missing is the setup of the database.

Jira suggest that whilst other databases are available, MySQL, MSSQL etc. their preferred DB is postgresql. Primarily because it’s common in their user space and support environment, meaning that their support and documentation is likely to be more readily available for postgresql instances than other DB’s.

Let’s follow the advice and install postgresql.

$ sudo apt-get install postgresql

At the time of writing this installs postgresql version 9.6 on Debian Stretch.

In order to create the environment that we can manage there are a couple of postgresql config changes that we make to ensure you can access the DB from another system – for managing with pgadmin 4.

Enable access to postgresql from specific network/IP addresses by editing pg_hba.conf under /etc/postgresql/9.6/main.

$ sudo vi /etc/postgresql/9.6/main/pg_hba.conf

Find the line:

host    all    all    127.0.0.1/32    md5

Add a line below to match your required IP addresses/subnets eg.

host    all    all    192.168.0.0/24  md5

This allows any machine with a 192.168.0.X address to access the DB.

Now we need to listen or bind to an IP address that is available on the network. By default postgresql only listens on 127.0.0.1 port 5432, meaning it will only accept connections to the local machine from the local machine.

$ sudo vi /etc/postgresql/9.6/main/postgressql.conf

Find the line beginning:

#listen_addresses = 'localhost'

Add a new line below it:

listen_addresses = '*'

Restart the postgresql service:

$ sudo systemctl restart postgresql.service

Databases and User

Create a database and a user for Jira/Confluence to use

$ sudo -u postgres createdb jira
$ sudo -u postgres createdb confluence
$ sudo -u postgres createuser jiradb

Set the users password and grant them access to the DB’s.

$ sudo -u postgres psql 
psql (9.6.10)
Type "help" for help.

postgres=# alter user jiradb with encrypted password 'mysupersecretpassword';

postgres=# grant all privileges on database jira to jiradb;

postgres=# grant all privileges on database confluence to jiradb;

When you install Jira and confluence you can then use the database settings you’ve just created.

Database Type: PostgreSQL
Hostname:      localhost
Port:          5432
Database:      jira
Username:      jiradb
Password:      mysupersecretpassword
Schema:        public

Selection_076

 

XSLT and SOAP —

XSLT and SOAP

All of our SOAP interactions with the Lagan CRM send and return SOAP and by association, XML. The normal practice of handling the sent or returned XML is by using XSLT to transform the data to and from the required format.

The forms product will submit XML through an XSL translation taking data from the POST’ed form data and turning it into the XML format/type required. The returned XML data must also be processed via an XSLT to present the data to the form.

How do we go about testing translations and stylesheets without constantly publishing forms and requesting data from the CRM server?

For this I used postman to submit and retrieve sample SOAP envelopes with the required XML soapenv:Body. Then I can take the returned sample data and save it to an XML file. Now I have a local sample of the XML I can use an XSLT tool to process it via a locally created stylesheet. No more repetitive form submissions or having to work with only the form product to develop the XSLT.

xlst_working

XSLT Tools

There are a very few XSLT tools that seem to do the job for free. Certainly when it comes to a GUI environment all the tools are paid for products.

At the command line there are some free options, but each have challenges. But I figured that just because it’s command line, doesn’t mean I can’t use it in a GUI. Atom has a very useful plugin that can be used to interface with the command line XSLT programs – atom-xsltransform. The settings for the plugin just point to the XSLT processor of your choice.

Once installed you press ctrl-shift-p whilst in your XML source file, it prompts you for the path of the XSLT transformation file to use and then returns the output into an edit tab in Atom.

MSXSL

For Windows I came across a very simple command line product from Microsoft MSXSL. It doesn’t look like there’s a recent version as this dates back to 2004. But as XML has been around for 20 years or so this may not be a problem. I did however find it seemed to produce broken output that looked to be to do with unicode. So maybe it’s not capable of handling the UTF-8 files I’m using.

xsltproc

This is from the world of Linux, but there is a port to Windows that works.

For Linux just install it from the repository:

$ sudo apt-get install xsltproc

For Windows, it’s harder work. Not significantly, but frustrating. You need to download a series of files, extract them all into the same place, to let their individual bin folders merge their contents. Then you can run the included xsltproc.exe and it should find all of the dll’s.

ftp://ftp.zlatkovic.com/libxml/

I chose the 64bit 7z files and extracted these files:

  • iconv-1.14-win32-x86_64.7z
  • libtool-2.4.6-win32-x86_64.7z
  • libxml2-2.9.3-win32-x86_64.7z
  • libxslt-1.1.28-win32-x86_64.7z
  • mingwrt-5.2.0-win32-x86_64.7z
  • openssl-1.0.2e-win32-x86_64.7z
  • xmlsec1-1.2.20-win32-x86_64.7z
  • zlib-1.2.8-win32-x86_64.7z

Saxon

This is a Java product and comes in a number of versions from home edition to professional that requires payment.

It’s hosted here on Sourceforge: http://saxon.sourceforge.net/

I downloaded the HE (home edition) and just placed the jar files somewhere I could use them.

From the Linux command line I used it like this:

$ java -jar saxon9he.jar -s:/home/user/lagan/xslt/FWTCaseFullDetails.xml -xsl:/home/user/lagan/xslt/FWTCaseFullDetails.xslt

Atom plugin settings

It’s a simple case of putting in the path of the executable you want to run. Pay attention to the order of the parameters for the tools. MSXML and xsltproc have the XML and XSL options in a different order.

For the Linux xsltproc settings I used:

/usr/bin/xsltproc %XML %XSL

For Saxon I had to be specific about where the jar file was as I haven’t installed it into the java class path.

java -jar /home/home/saxon/saxon9he.jar -s:%XML -xsl:%XSL

Stylesheets

The XSLT stylesheet acts as the instruction set to take the XML input and apply the XSLT logic to transform the XML content into another format such as text or HTML.

W3Schools has some useful guidance here: https://www.w3schools.com/xml/xsl_intro.asp

Another useful intro: https://www.tutorialspoint.com/xslt/

 

VOF – Accessing H2 from Another System — August 30, 2018

VOF – Accessing H2 from Another System

Following on from Verint Online Forms using H2 seems pretty straight forward locally. It fires up a web server and you can manage the H2 database straight from there.

You need the VOF database details you put into Config.sh then you can start connecting to it from within the browser.

eg.

JDBC URL: jdbc:h2:~/lagan/dform-x.x.x/db/kana-integration/h2

But if you’re running dforms on your virtual box development server you’ll be denied because the setting webAllowOthers is not set.*

This is easily remedied and still secure as your virtual box should be using a “host only network adapter” so only your system can get to it.

Create a file in you home folder and put the following one line into it:

$ vi ~/.h2.server.properties

webAllowOthers=true

That’s all. Once the server runs it will probably add some more to the file, but you should now be able to access the H2 GUI at http://192.168.56.2:8082

The other thing that may be stopping you is by default H2 will look at your systems name and resolve it to an IP. This is the IP that will listen on port 8082. If you’ve setup a virtual box then your /etc/hosts file may contain a line like 127.0.1.1 debian and this will be an inaccessible IP. You’ll need to change this so you have an entry matching your machines DNS name eg.

192.168.56.2    debian debian.domain.local

When you start H2 you should then see it listen on the correct address.

$ sh ./h2.sh
Failed to start a browser to open the URL http://192.168.56.2:8082: Browser detection failed and system property h2.browser not set

Don’t worry about the error. It’s complaining because we’re running a headless non-Windowed server, there is no X11 and there is no browser to launch. The important thing is it’s starting on the right IP address.

Other Useful Options

By default H2 tries to start a browser as shown in the above error message. You can stop this behaviour by passing parameters to the h2.sh call

-web – start a web service
-tcp – start a tcp service
-pg – start a postgres service
-browser – try and start a browser

So if you just want a web service

$ sh ./h2.sh -web

Web Console server running at http://192.168.56.2:8082 (others can connect)

No more error message!

You can chain them too eg. sh ./h2.sh -web -tcp -pg or for a more permanent solution edit the h2.sh file and add -web to the java line:

java -cp "$dir/h2-1.4.197.jar:$H2DRIVERS:$CLASSPATH" org.h2.tools.Console "$@" -web

 

* Yes I know you can set this in the Web GUI. But if you can’t get to the Web GUI (the whole point of this article) you’ll need to set it from the servers command line.

Verint Online Forms —

Verint Online Forms

We’re new to this and trying to integrate a form solution with our Lagan CRM system. We have a corporately installed test and production system for forms, but it get frequent usage by many non-IT related staff, so I thought about deploying our own dev system.

The forms products are pretty much Jetty programs with a database requirement. Looking at the config files for the initial deployment package they are looking for either H2, Oracle or MSSQL. That means our only real dev option is H2.

Step 1 – Install the H2 database

Download the platform independent zip from http://www.h2database.com/html/main.html and extract it into a suitable location.

Make sure you have a $JAVA_HOME environment variable set, and ensure you have a Java JDK (not just a JRE) installed. On my Debian system I set it to the default java instance (which just happens to be Java 10):

$ export JAVA_HOME=/usr/lib/jvm/default-java

Then run the H2 program:

$ cd h2/bin
$ sh ./h2.sh

This fires up a browser session and gives you an icon in the tray if you’re running a windowed environment.

Step 2 – Install dforms

Extract the dfoms zip file.

Edit the config.sh file in dforms-x.x.x/bin as necessary. The only changes I made to this one was the jdbc user and password. I prefer not to use defaults.

Run the program:

$ cd dforms-x.x.x/bin
$ sh ./Run.sh

We then have a running dforms program listening on the default port 9081. You can use your browser to visit it at http://localhost:9081/auth/login and logon using the defaults Admin/Admin credentials.

Step 3 – Install dforms-leadapter

Extract the dfoms-leadapter-x.x.x zip file.

Edit the config.sh file in dforms-leadapter-x.x.x/bin as necessary. I made a few more changes to this one, again the jdbc user and password – it is a different database than the dforms one and there are two of them in this config. But also the flweb_lagan_uri and flweb_user and flweb_password to match our environment.

Run the program:

$ cd dforms-leadapter-x.x.x/bin
$ sh ./Run.sh

We then have a running dforms-leadapter program listening on the default port 9082. You can use your browser to visit it at http://localhost:9082/auth/login and logon using the defaults Admin/Admin credentials.

When is a Question Mark not a ? — August 29, 2018

When is a Question Mark not a ?

That’s a morning of smashing my face on the desk again. I deployed my dev program onto a production system and then started crying as it stopped working as it should.

It seemed that none of my query string parameters were making it through to the controller. I called up some debugging and dumped out my $request and $request->all() etc. and discovered that the parameters although shown in the browser dev window went AWOL between server and controller. On my dev environment it all acted as it should.

So there must be something different. PHP v7.2 on dev and v7.0 or production maybe? No, much simpler than that. None of the Laracasts and Laravel related Googling pulled up any particular clues. It wasn’t until I looked at Nginx and parameters not being passed to PHP that I got a hit.

https://serverfault.com/questions/685525/nginx-php-fpm-query-parameters-wont-be-passed-to-php

The answer was as simple as adding in the $is_args into my Nginx virtual server config.

location / {
  try_files %uri $uri/ /index.php$is_args$query_string;
}

Up until now I guess I’ve been using routing with the parameters as part of the URI. Now I’m using some query string parameters I need to put in the ?, which is the $is_args variable.

So why not a problem in dev? Because I’m not using Nginx, I just use artisan serve to debug my development program.

Postman is Awesome — August 21, 2018

Postman is Awesome

I’m in the process of testing and documenting my API’s and up until yesterday I’d only used postman to test my responses gave me something back. But it’s capable of so much more!

I can use it to generate the documentation for each call I make, publish it online and share it with colleagues.

But I can also use it to carry out unit tests on my API calls. Which is awesome.

Then I find I can use globals and environment variables to migrate my tests and documentation between development and production systems… which is even more awesome.

Continue reading

Postman Makes API Development Simple — August 20, 2018
Laravel and Lagan Web Services – SOAP API — August 17, 2018

Laravel and Lagan Web Services – SOAP API

SOAP is a dirty word to me. But I have a need to interact with our CRM system to import / extract data. My go to platform for most of my PHP work is Laravel. So I looked at interacting with Lagan CRM using SOAP calls from PHP.

I started off accessing the Lagan WSDL pages to see what the capabilities of the API are.

http://[laganserver:8080]/lagan/services/FL?WSDL
http://[laganserver:8080]/lagan/services/FLAuth?WSDL
http://[laganserver:8080]/lagan/schema/FLService.wsdl

Now I can see the self documenting API calls I can make. I just need to create the SOAP envelope to pass data to the service with the call I want to make.

Continue reading

Sudo and Proxy / Environment Settings — August 15, 2018

Sudo and Proxy / Environment Settings

When you run a program using sudo what tends to happen is the sudo/root account fails to do anything useful on the internet. It times out trying to connect to systems to download updates that are required by elevated permissions.

We discovered using sudo composer self-update failed to update the core instance of composer, not because of permissions, but because it could not get to the internet to download it.

Set the environment variables that get persisted within your /etc/sudoers file by running:

$ sudo visudo

Seach for the line

Defaults    env_reset

and change it to

Defaults    env_keep += "ftp_proxy http_proxy https_proxy no_proxy"

Now your proxy will be set within your sudo environment too.

References

Source: https://stackoverflow.com/questions/8633461/how-to-keep-environment-variables-when-using-sudo

Laravel API and Bootstrap Form Validation — August 13, 2018

Laravel API and Bootstrap Form Validation

This caused me some grief today. I spent the day adding validation rules into my Laravel resource controller and rather foolishly set HTML5 validation parameters on my Vue.js / Bootstrap 4 form component.

Why foolishly?

Well if you follow the Bootstrap 4 JavaScript function to call form.checkValidity() you’re actually calling the HTML5 built in function. Not a Bootstrap function as I originally thought.

When Laravel validation failed at the resource controller and it pushes back 422 (Unprocessable Entity) and a json error object:

{"message":"The given data was invalid.","errors":{"name":["The name field is required."]}}

I thought Bootstrap was seeing the Laravel validation errors and flagging up fields as not valid. So I could not understand why one of my fields didn’t show as invalid when according to Laravel it was!

What I was actually doing was HTML5 validation and ignoring my Laravel validation response all together. With the API it’s best NOT to try to use both HTML5 and Laravel validation. You’ll get a confusing UX that uses a mix of browser error messages/popups and Bootstrap CSS error handling.

Make sure you add novalidate to your form tag – this ensures HTML5 browser validation is prevented at the form level.

To resolve the Laravel validation part I just use the Laravel json error object and DON’T USE checkValidity(), my axios .catch(error) processes the Laravel errors by calling showErrors(error.response.data)

 this.axios.post('/api/v1/mycall/',
   this.data
 ).then(() => {
   // That worked out well, do something.
 }).catch(error => {
   this.showErrors(error.response.data)
 })

 showErrors: function (error) {
  Object.keys(error.errors).forEach((field) => {
    let input = document.getElementById(field)
    input.classList.add('is-invalid') // Bootstrap invalid form input
  })
 }

This iterates through the json errors and adds the class is-invalid to the fields that Laravel tells me are invalid. This triggers Bootstraps CSS to show the field with a red border and unhides the form-control subsequent/child div that has a class of invalid-feedback

References

https://getbootstrap.com/docs/4.1/components/forms/#validation

When using Vue.js and Laravels json response the actual usage is closer to the server-side examples:

https://getbootstrap.com/docs/4.1/components/forms/#server-side

Linting — August 6, 2018

Linting

No, this isn’t about taking the fluff from your belly button – but you’re close.

https://eslint.org/

Linting your written code is a method of ensuring that it meets consistent syntax and style guidelines. Eg. ensuring you indent function blocks by 4 spaces and not tabs, placing curly braces {} on new lines, having spaces following function names and parameters etc.

JavaScript, being a dynamic and loosely-typed language, is especially prone to developer error. Without the benefit of a compilation process, JavaScript code is typically executed in order to find syntax or other errors. Linting tools like ESLint allow developers to discover problems with their JavaScript code without executing it.

ESLint is a great way of ensuring my (ECMA)JavaScript coding style is consistent and correct. Using it within atom.io is a great way to keep my work tidy. Just add the ESLint package and we’re set to track and tidy my .js files.

But I want the same rules for my .vue files. I found this really handy: https://alligator.io/vuejs/vue-eslint-plugin/

Add the atom package linter-eslint then go to its settings button.

Find the option “List of scopes to run ESLint on…” and add onto it:

text.html.vue

So in my case it becomes:

source.js, source.jsx, source.js.jsx, source.babel, source.js-semantic, text.html.vue